4/18/25: GS doctoral final defense

Pass! Congratulations!

PhD Dissertation Defense 

Friday, April 18 2025 3:00pm (CST)

(7:00am Sydney Australia)

Location

 https://ualr-edu.zoom.us/j/8375549395

"OPTIMIZING SMALL AI MODELS FOR BIOMEDICAL TASKS THROUGH EFFICIENT KNOWLEDGE TRANSFER FROM LARGE DOMAIN MODELS"

by:

 Girish Sundaram

Doctoral Candidate

Information Science

DCSTEM, UA Little Rock 

Advisor: Dr. Daniel Berleant

ABSTRACT

The PICO (Population, Intervention, Comparison, Outcome) framework is a widely adopted methodology for structuring clinical research questions and extracting relevant information from unstructured medical texts. However, traditional approaches for PICO classification demand computationally expensive domain-specific language models, such as BioBERT and ClinicalBERT, which require extensive training and large annotated datasets.

This dissertation introduces Distilled Rapid Embedding Transfer (DRET), a novel knowledge transfer method designed to enable resource-constrained domain adaptation. DRET aims to efficiently transfer biomedical domain knowledge from large, specialized models to a compact, general-purpose model, DistilBERT, thereby enhancing its ability to perform domain-specific tasks without access to the original training datasets. This study hypothesizes that domain knowledge transfer can be achieved without significant computational overhead and that general-purpose models can attain performance levels comparable to domain-specific models in PICO classification tasks.

To validate this hypothesis, a systematic literature review was conducted to establish prior work on domain adaptation in biomedical NLP. The experimental framework involved training and evaluating variations of general-purpose and domain-specific models on the EBM-NLP dataset, a class-imbalanced corpus of ~5,000 clinical trial abstracts. (Bepnye, n.d.) The effectiveness of knowledge transfer was quantitatively assessed using evaluation metrics such as balanced accuracy, Matthews correlation coefficient (MCC), F1-score, precision, recall, GMean, and average PR AUC. Additionally, cosine similarity, nearest neighbor analysis, cluster coherence (silhouette score), and semantic shift analysis were employed to measure domain understanding improvements.

The results indicate that DRET significantly enhances DistilBERT’s performance on PICO classification, achieving competitive results against domain-specific models while requiring a fraction of the computational resources. Class-wise performance analysis revealed that knowledge transfer benefits were particularly pronounced in the Intervention and Outcome categories, where DistilBERT exhibited higher recall and precision. Furthermore, DRET reduces reliance on large-scale retraining, making domain adaptation feasible in resource-constrained settings.

The findings contribute to the growing field of biomedical NLP by demonstrating that efficient domain adaptation can be achieved through strategic embeddings transfer, a type of knowledge transfer, bridging the gap between large-scale domain-specific models and smaller general purpose AI models that meet real-world deployment constraints. This research has implications for accelerating AI-driven systematic literature reviews, improving clinical decision-support systems, and enabling scalable biomedical text mining applications. Future work will explore extending DRET to additional biomedical tasks, further refining its embedding consolidation strategy, and integrating external domain-specific knowledge sources to enhance performance.