Differentiable multi-agent actor-critic for multi-step radiology report summarization system
Inventors
Kumar Karn, Sanjeev • Farri, Oladimeji
Assignees
Publication Number
US-12362049-B2
Publication Date
2025-07-15
Expiration Date
2042-05-04
Interested in licensing this patent?
MTEC can help explore whether this patent might be available for licensing for your application.
Abstract
Systems and methods for using a differentiable multi-agent Actor-Critic (DiMAC) for multi-step radiology report summarization. The tasks of extracting salient sentences and phrases are divided across two collaborating agents that are trained end-to-end using reinforcement learning (RL).
Core Innovation
The invention provides a system and method for multi-step summarization of radiology reports using a differentiable multi-agent Actor-Critic (DiMAC) framework. The system includes a word extraction network that extracts salient words from the FINDINGS section of a radiology report, a sentence extraction network that extracts one or more salient sentences based at least in part on the extracted words, and an abstractor network that condenses these sentences into a concise IMPRESSIONS summary. These networks are trained end-to-end via multi-agent reinforcement learning, leveraging a critic network to estimate value functions used in policy gradient computations and a communication channel passing gradient information between the word and sentence extraction networks.
The problem addressed is the challenge in automatic abstractive summarization of information-rich radiology reports, specifically generating accurate and comprehensive IMPRESSIONS summaries from lengthy FINDINGS sections. Prior single-step and two-step approaches with single extractive agents suffer from lacking explainability, requiring large labeled datasets, and risks of extracting irrelevant or missing salient content, leading to incomplete or incorrect summaries. This invention overcomes these issues by employing a multi-agent reinforcement learning system with coordinated word and sentence extraction agents that communicate during training to enhance extraction precision and summarization quality.
The system uses bi-directional LSTM-based encoders for words and sentences and pointer generator networks for abstractive summarization. During training, the word and sentence extraction networks act as independent agents that communicate via a sigmoidal mechanism, allowing the system to learn policies jointly through individualized rewards based on ROUGE L recall metrics for sentences and keyword presence for words. A centralized critic conditions on all agent actions and states to compute policy gradients that update agents cooperatively. Results demonstrate improved summary coverage, accuracy, and explainability compared to prior single-agent or non-communicative systems.
Claims Coverage
The patent includes three independent claims covering a system and methods for multi-step radiology report summarization using coordinated word and sentence extraction networks trained end-to-end with a critic and communication channel.
Multi-step radiology report summarization system with coordinated extractors
A system comprising a word extraction network to extract words from a radiology report, a sentence extraction network to extract salient sentences based on extracted words, and an abstractor network to condense the sentences into a concise summary. The networks are trained end-to-end using a critic estimating a value function for policy gradient calculation and a communication channel passing gradient information between the word and sentence extraction networks.
Training method using multi-agent reinforcement learning with communication
A method for configuring the system by iteratively inputting training data from a FINDINGS section, computing states, sampling actions where only one agent acts at a time while the other pauses, communicating actions via a channel generating a sigmoidal value fed to one extractor and its complement fed to the other, computing individual rewards, and updating the sentence extraction, word extraction, and critic networks accordingly.
Automatic summarization method for radiology report FINDINGS section
A method comprising acquiring a radiology report, inputting its FINDINGS section to an automatic summarization system including at least a word extraction network, a sentence extraction network, a communication channel, and an abstractor trained via differential Multi-agent Actor-Critic reinforcement learning, outputting an IMPRESSIONS section, and presenting it to a user.
The claims collectively cover a coordinated multi-agent reinforcement learning framework for extractive and abstractive multi-step summarization of radiology reports, detailing the system architecture, training methods, communication mechanisms, and application procedures to generate concise IMPRESSIONS from FINDINGS sections.
Stated Advantages
The DiMAC system provides more precise and accurate radiology report summaries compared to single-step or single-agent two-step approaches.
The multi-agent approach improves explainability by using salient keywords and sentences, enhancing the basis for summarization.
End-to-end training with communication between agents yields richer training signals, minimizing learning effort and improving policy learning.
The generated IMPRESSIONS sections reflect human-level inference and actionable information supporting workflow efficiency and better-informed clinical diagnosis.
The system can operate with limited labeled data and reduces errors compared to prior methods.
Documented Applications
Automatically generating concise summaries (IMPRESSIONS sections) from the FINDINGS section of radiology reports to assist referring physicians in clinical diagnosis and workflow.
Use in medical imaging systems to produce radiology report summaries that aid in explaining, confirming, or excluding differential diagnoses.
Integration in medical imaging modalities (e.g., CT, MR, PET, SPECT, ultrasound, x-ray) and associated medical systems for real-time or post-procedure report summarization.
Deployment in server or standalone systems for processing stored radiology reports from databases or picture archival and communication systems.
Interested in licensing this patent?