Research you can
look into.
Build and evaluate a system that extracts information from publications using predefined parameters, returning either the requested data or “NR” when information is not reported or cannot be located.
The work brought together giles® AI, R-S-S and academic expertise. It evaluated structured extraction from atopic dermatitis publications using Google Gemini 2.5 Pro, with the fields supplied through a CSV template at run time.
Atopic dermatitis · 11 evaluation articles + 6 supplements
How the evaluation worked.
- 01
Define the extraction task
The source collection included 13 publications and six supplementary materials. Fields covered treatment arms, populations, disease severity, assessment timepoints and EASI and IGA outcomes.
- 02
Develop and refine the template
Two publications were used during development and validation. The table headings were refined over three iterations before applying the template to the remaining 11 publications.
- 03
Assess reported and missing data
The final evaluation covered 11 articles and six supplements. Of 2,118 extracted data points, 1,266 had reported data and 852 were not reported.
Accuracy includes knowing when data are missing.
The poster reports averages of 95% overall accuracy, 94% reported-data accuracy, 95% precision for “not reported” entries and 98% recall for “not reported” entries across the evaluation sources.
What the measures mean
Was each extraction cell filled correctly, with a value or “NR”?
When a source reported data, was the value extracted correctly?
When the system returned “NR”, was the information actually unreported?
When information was unreported, did the system correctly return “NR”?
Read the results in context.
The 98% figure measures recognition of information that was not reported; it is not a general measure of finding all relevant research evidence. Results are specific to the documents, extraction fields and evaluation method described in the poster.
Explore the full publication