NLP Project Plan: ADR Detection in Medical Literature

Problem Statement

Detect and extract structured adverse drug reaction (ADR) mentions from ~500 biomedical paper abstracts (currently in PDF/mixed format), producing typed drug-ADR pairs suitable for ingestion into a pharmacovigilance database.

Pipeline Overview

PDF/mixed source
  └─> [Phase 0] Text extraction (Docling / pdfplumber + spacy-layout)
        └─> [1] DRUG NER  (scispaCy BC5CDR baseline → fine-tune if needed)
        └─> [2] ADVERSE_REACTION NER  (same model, separate eval pass)
              └─> [3] Relation linking (sentence co-occurrence + negation → NLI reframe if needed)
                    └─> [4] Normalization (DrugBank/RxNorm, MedDRA) — business logic, not ML

Data Strategy

• ~500 abstracts in PDF/mixed formats — text extraction required before NLP work
• Nothing annotated yet
• Plan: LLM pre-annotation (GPT-4 via spacy-llm) → expert correction in Prodigy
• Set aside ~100 abstracts as held-out eval set before any annotation begins
• Target: ~300–400 annotated abstracts for NER training

Annotation Plan

Schema

• DRUG — drug/substance mention spans (generic, not role-specific)
• ADVERSE_REACTION — adverse reaction term spans (generic)
• Relations: drug-reaction pairs (separate pass after NER is stable)

Schema principles

• Do NOT use composite labels like "DRUG_CAUSING_ADR" — couple NER with role assignment is a mistake
• Normalization to DrugBank / MedDRA is business logic downstream, not part of the annotation schema

Workflow

1. Pilot: 50–100 abstracts, domain expert annotates DRUG spans only (correction of BC5CDR predictions)
2. Evaluate BC5CDR baseline on same set — decide how much fine-tuning is needed
3. ADVERSE_REACTION pass (single-label correction mode)
4. Relation pass: link drug-reaction pairs within sentences
5. Train first model after pilot, run training curve, validate learnability before scaling

Team

• 1–2 domain experts (clinicians / pharma researchers) — ideal small expert team
• Prodigy on Ellf cluster for annotation

Evaluation Strategy

• Hold out ~100 abstracts as a fixed test set before training starts — never train on these
• Evaluate NER with token-level and span-level F1, split by entity type
• Rule-of-10: distinguish 85% vs 90% accuracy needs ~1000 span-level eval examples — plan accordingly
• Relation evaluation: precision/recall on drug-ADR pairs (exact match: both spans + link correct)
• Run training curves after pilot to decide whether more annotation helps or schema needs revision
• Baseline: evaluate scispaCy BC5CDR out-of-the-box before annotating anything

Roadmap

Phase 0 — Text extraction (prerequisite)

• Extract clean abstract text from PDFs
• Tools: Docling, pdfplumber, or spacy-layout
• Output: plain text files per abstract, ready for annotation

Phase 1 — Baseline evaluation

• Run scispaCy en_ner_bc5cdr_md on 50 sample abstracts
• Measure what it gets right on DRUG and ADVERSE_REACTION before committing annotation budget

Phase 2 — NER annotation + model

• LLM pre-annotation → expert correction in Prodigy (ner.correct)
• Single-label passes: DRUG first, ADVERSE_REACTION second
• Train after pilot, run training curve, scale only if curve is still rising

Phase 3 — Relation extraction

• Start with rule-based co-occurrence + negation detection as baseline
• Escalate to NLI reframe ("drug X caused reaction Y" → entailed/neutral/contradicted) if needed

Phase 4 — Normalization (business logic)

• Map DRUG spans to DrugBank / RxNorm
• Map ADVERSE_REACTION spans to MedDRA
• This is a lookup layer, not a model — keep it separate from NLP components

Next Steps

1. Solve text extraction (Phase 0) — this is a blocker for everything else
2. Evaluate scispaCy BC5CDR on a sample before annotating
3. Set up Prodigy annotation task on Ellf with LLM-assisted pre-annotation