Serverless data ingestion for transforming and delivering data in system endpoints
Inventors
Assignees
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Publication Number
US-12353378-B2
Publication Date
2025-07-08
Expiration Date
Abstract
There are provided systems and methods for serverless data ingestion for transforming and delivering data in system endpoints. An entity, such as company or business, may utilize computing services provided by a service provider. When providing these services, one or more computing services, processors, or the like of the service provider's computing architecture may be used. This may include use of serverless computing environment to delivery data to internal and/or external endpoints. To automate data delivery in a serverless environment, computing code for data processing and delivery pipelines may be configured and deployed in the serverless environment. This allows applications and microservices to be executed in the serverless environment to provide a computing architecture for without requiring servers and designation of server clusters. The data pipeline may provide operations for ingesting, transforming, and loading data when received for delivery to endpoints.
Core Innovation
Upon startup of an application executing in a serverless computing environment, the system loads data processing configurations for a data processing pipeline that processes and delivers data to system endpoints. The application creates a plurality of processing threads that include at least one data ingestion thread, at least one data transformation thread, and at least one data loading thread, and the processing threads are created in reverse order such that the at least one data loading thread is created before the at least one data transformation thread and the at least one data transformation thread is created before the at least one data ingestion thread.
Using the at least one data ingestion thread, the system ingests data received from a first system endpoint, and the at least one data transformation thread transforms the ingested data into a data format utilized by a second system endpoint designated to receive the data. After transformation, the system loads the transformed data from the at least one data transformation thread to the second system endpoint using the at least one data loading thread.
The data processing configurations include configurable data transformation mechanisms, operators, and inspectors. Inspectors inspect data using coded expressions and route the data to system endpoints through a data processing pipeline, and configurable and pre-coded pipeline structures transform identified data into endpoint-utilized data formats, including user-defined success or failure criteria for evaluating the coded configurations.
Claims Coverage
The independent claims are directed to a system, a method, and a non-transitory machine-readable medium. They share 3 inventive features: serverless startup configuration loading, reverse-order creation of ingestion/transformation/loading threads, and endpoint-specific transformation and delivery.
Loading data processing configurations upon application startup in a serverless computing environment
Upon startup of an application executing in a serverless computing environment, loading data processing configurations for a data processing pipeline via which data is processed and delivered to system endpoints.
Reverse order processing thread creation for ingest, transform, and load
Creating a plurality of processing threads for the application, wherein the plurality of processing threads includes at least one data ingestion thread, at least one data transformation thread, and at least one data loading thread, and wherein the plurality of processing threads are created in reverse order such that the at least one data loading thread is created before the at least one data transformation thread and the at least one data transformation thread is created before the at least one data ingestion thread.
Endpoint-to-endpoint ingestion, transformation into endpoint-utilized data format, and loading
Ingesting data received from a first one of the system endpoints using the at least one data ingestion thread, transforming the data ingested by the at least one data ingestion thread using the at least one data transformation thread into a data format utilized by a second one of the system endpoints designated to receive the data, and loading the transformed data from the at least one data transformation thread to the second one of the system endpoints using the at least one data loading thread.
The independent claims consistently cover serverless startup configuration loading, reverse-order thread creation for ingestion, transformation, and loading, and transformation of ingested data into a format utilized by a designated receiving endpoint followed by delivery to that endpoint.
Stated Advantages
Documented Applications
No documented applications found
Interested in licensing this patent?