Document crawling systems and methods

Inventors

Wagers, Doug R.

Assignees

Softek Illuminate Inc

Abstract

Systems and methods are provided for crawling and indexing documents stored in a data storage system. A crawler system processes multiple jobs that each correspond to crawling documents in the data storage system. Each job includes priority data and crawling instructions. The crawler system stores each job in a priority queue in a sequence based on the priority data. The crawler system assigns each job in the priority queue to a next available processing module for processing based on the stored sequence. Before processing each job, the crawler system determines whether to segment the job into smaller steps based on the corresponding crawling instructions. If the job is segmented, one of smaller steps is processed to crawl a group of the documents in the data storage system. The remaining steps are stored in the priority queue to wait for processing.

Core Innovation

The disclosure relates to a document crawling and indexing system in which a data management application crawls documents in a data storage system by scheduling crawl job modules. A scheduling module retrieves a plurality of job modules from a data store, where each job module includes corresponding crawling instructions and corresponding priority data for crawling documents. The job modules are stored in a priority queue in a sequence based on the corresponding priority data, and an execution module assigns each job module to one of a plurality of processing modules according to the sequence for processing.

Each assigned job module identifies a step for processing based on the corresponding crawling instructions, where the step comprises crawling a group of the documents. The job module processes the step to crawl the group of the documents in the data storage system, and then determines if at least one additional step for processing is required based on the corresponding crawling instructions. When an additional step is required, the at least one additional step comprises crawling another group of the documents.

After processing determines that at least one additional step for processing is required, the job module is rescheduled to the scheduling module for insertion into the priority queue. The disclosure further describes crawling instructions that can segment crawling into additional steps, where a portion of crawling is executed and remaining steps are rescheduled back into the priority queue. The scheduling behavior is motivated to reduce impact on the system being crawled, prevent job starvation, and limit bandwidth and throughput bottlenecks, while also supporting priority determination based on document type, creation/edit age, job size, status data, and recurrence intervals.

Claims Coverage

The independent claims present four inventive features in a job-module based crawling workflow with priority-queue ordering, step-based document-group crawling, iterative rescheduling, and, in one claim, status-data-based step identification.

Across the independent claims, the core inventive coverage is a closed-loop crawling workflow that retrieves job modules containing crawling instructions and priority data, orders the job modules in a priority queue, assigns job modules to processing modules based on the priority sequence, executes step-based crawling of groups of documents, determines whether additional steps are required, and reschedules job modules back into the priority queue. One independent claim further incorporates status data into step identification.

Stated Advantages

Documented Applications