Asynchronous document ingestion and enrichment system

Inventors

Dellinger, Nathan • Peters, David

Abstract

Provided are asynchronous data ingestion and enrichment systems and methods. The systems comprise a plurality of components (e.g., ingestion components, enrichment components, and/or publishing components). Instead of passing data from one component to another, the data is sent to a messaging queue that formats and hold the data until the subsequent component is ready to receive it. Additionally, each component comprises a central microservice and a plurality of instances, the central microservice configured to communicate with each instance of the plurality of instances.

Core Innovation

The invention provides an asynchronous data ingestion and enrichment system that instantiates a plurality of ingestion software processing modules, a plurality of enrichment software processing modules, and a plurality of publishing software processing modules. Each ingestion module includes an ingestion central microservice configured to manage a plurality of ingestion instances, each enrichment module includes an enrichment central microservice configured to manage a plurality of enrichment instances, and each publishing module includes a publishing central microservice configured to manage a plurality of publishing instances.

The architecture instructs ingestion modules to publish to one or more messaging queues and instructs enrichment modules to consume from the messaging queues and publish to the messaging queues. The publishing software processing modules consume from one or more of the plurality of messaging queues, completing the ingestion-enrichment-publishing pipeline under the coordinated microservice management.

The system applies a de-duplication algorithm in at least one ingestion software processing module. Applying the de-duplication algorithm comprises transmitting a message from the enrichment central microservice to an Application Processing Interface (API) to determine whether a document has been previously processed by the system, and further comprises the enrichment central microservice calling back to at least one other component of the system to determine whether the document has been previously processed.

The enrichment software processing modules are configured to support failover/resume without duplication of work. In response to a first enrichment instance ceasing operation, a second enrichment instance resumes operation such that there is no duplication of work.

Claims Coverage

This partial set includes two independent claims, clm-00001 and clm-00013, both directed to the same asynchronous ingestion-enrichment-publishing architecture with messaging queues, centralized microservice management, de-duplication via an API/callback flow, and enrichment failover/resume without duplication of work. The claims cover four inventive features.

Across both independent claims, the main inventive elements are the coordinated instantiation and microservice management of ingestion, enrichment, and publishing instances, asynchronous message-queue communication, de-duplication based on whether a document has been previously processed using an enrichment-central-microservice API and callback flow, and enrichment failover/resume that avoids duplicate work.

Stated Advantages

Documented Applications