Adding a Clinical Focus to Evaulating Multi-Modal Data Representations

“How to validate synthetic text generated from images for healthcare applications.”

Figure 1: Proposed clinical workflow applications of ML to radiology - using the CXR workflow as an exmaple. [CDSS = clinical decision support system, CXR = Chect x-ray, EHR = Electronnic health record, PACS = Picture archiving and communication system]

The use of Natural Laguage Generation (NLG) for the auto generation of radiology reports has the potential to provide multiple radiology workflow applications. Free text reports pose a challenging task from which to compare NLG outputs due to the “ambiguity, syntax, synonymy, medical abbreviations”, use of negation, reference to “out of reach” information, linking of associated findings, and overall individual variation in reporting style seen between different radiologists

An series of evaulation techniques were tested cobining components from machine translation metrics (Bleu, Rougue, Meteor), Clinical metrics (CheXpert, Mirqi), and a suggested Clinical Scoring System.

Results

Combinations of evaulation metrics were tested against three experiemnts which were trained on single sentence descriptions, multiple sentence descriptions and including the “impression and findings” section of the reports alongside the multiple sentence descriptions.

The metrics used to evaluate the performance of models for clinical tasks require further refinement to ensure clinical accuracy is captured. The effect on model performance from adapting model training as well as performance on external dataset was also conducted.

Three potential uses for NLG models in the clinical radiological workflow highlighted in this work include;

• use as a safety-net for radiologists to auto-fill positive findings if not included in the report by the radiologist,
• provide preliminary reports for acute CXRs to support junior doctors interpreting scans on the wards in the first instance whilst awaiting the radiologists report communicating critical findings,
• automate follow up oncology scans, e.g. CT, reporting to provide a faster indication if a malignancy has progressed / quantifying response to therapy.

Output	Link
Open Source Code & Documentation	Github
Case Study	Awaiting Sign-Off
Technical report	Here