NLP Project Plan: Fraud Report Classifier

Problem Statement

Build a pipeline that processes analyst-written fraud investigation summaries and produces four outputs: fraud type (multi-class, 6 labels), affected product, urgency level, and legal escalation flag. Urgency and legal escalation are derived from business rules applied downstream of model predictions — not model outputs themselves. Starting from scratch on annotation.

Pipeline Overview

Analyst report text

│─→ [Fraud Type Classifier] ← supervised textcat, 6-class exclusive

|─→ [Social Engineering Detector] ← supervised textcat, binary

|─→ [Product Extractor] ← rules (PhraseMatcher on known product names)

|─→ [Amount Extractor] ← regex / MONEY NER for dollar amounts

|─→ [Business Logic Layer] ← NOT a model

|─→ urgency: fraud_type × amount × product → policy thresholds

|─→ escalate: urgency == HIGH → legal team routing

Key architecture decision: Urgency and legal escalation are policy decisions, not language-understanding tasks. Internal thresholds ($100K, insider fraud type) are encoded as rules owned by the fraud team, not embedded in model weights.

Components

Data Strategy

• Source: Analyst-written fraud investigation summaries (free text, high quality, domain-consistent language)
• No existing labels — annotating from scratch
• insider_fraud and insurance_fraud estimated at 10–15% each — random sampling sufficient

Annotation Plan

Fraud type classifier

• Recipe: textcat.correct with LLM pre-annotation
• Labels: account_takeover, money_laundering, card_fraud, application_fraud, insider_fraud, insurance_fraud (exclusive)
• Target volume: 400–500 annotated examples
• Evaluation split: Set aside ~100 examples before annotation starts (document-level split)
• Pilot first: Annotate 50–75 manually (no LLM). Fix schema before scaling.

Social engineering vector

• Recipe: textcat.binary
• Label: social_engineering_vector (true/false)
• Pass: Second pass after fraud type annotation is stable
• Target volume: 200–300 examples

Schema decision

• social_engineering is a method, not an outcome — removed from fraud type labels
• Added as a separate binary attribute to avoid confusable label pairs

Evaluation Strategy

Test set

• Hold out ~100 examples before any annotation begins (document-level split)
• Never used in training; kept constant as the ground truth benchmark

Metrics

• Per-class F1 for all 6 fraud types — do not rely on macro average alone
• insider_fraud and insurance_fraud tracked separately
• Confusion matrix — expected confusables: card_fraud ↔ account_takeover, application_fraud ↔ money_laundering

Baselines

• Most-frequent-class baseline before any model evaluation

Training curves

• Train on 25/50/75/100% after each annotation batch
• Rising at 100% → annotate more; flat → investigate schema or architecture

Memorisation check

• Train on pilot examples, evaluate on those same examples — must be near-perfect

Roadmap

Next Steps

1. Hold out ~100 docs as the test set before touching any annotation tool
2. Read 50–75 reports manually to validate the schema
3. Write one-page annotation guidelines with label definitions and tiebreaker rules
4. Annotate the pilot batch using the textcat recipe
5. Train a quick baseline and run the memorisation check
6. If training curves are rising, scale with LLM-assisted annotation
7. Build the rules layer (product PhraseMatcher, amount regex) in parallel

Decision Log

• Urgency: Business logic. Thresholds ($100K, insider fraud type) are policy rules.
• Legal escalation: Business logic derived from urgency. Policy-owned.
• Product extraction: PhraseMatcher rules first. Analyst reports name products explicitly.
• Fraud type: Supervised textcat, 6-class exclusive. Core NLP task.
• Social engineering: Binary vector flag, not a fraud type label. It's a method, not an outcome.
• Evaluation-first: 100-doc test set held out before annotation begins.

NLP Project Plan: ADR Detection in Medical Literature

Problem Statement

Detect and extract structured adverse drug reaction (ADR) mentions from ~500 biomedical paper abstracts (currently in PDF/mixed format), producing typed drug-ADR pairs suitable for ingestion into a pharmacovigilance database.

Pipeline Overview

PDF/mixed source
  └─> [Phase 0] Text extraction (Docling / pdfplumber + spacy-layout)
        └─> [1] DRUG NER  (scispaCy BC5CDR baseline → fine-tune if needed)
        └─> [2] ADVERSE_REACTION NER  (same model, separate eval pass)
              └─> [3] Relation linking (sentence co-occurrence + negation → NLI reframe if needed)
                    └─> [4] Normalization (DrugBank/RxNorm, MedDRA) — business logic, not ML

Data Strategy

• ~500 abstracts in PDF/mixed formats — text extraction required before NLP work
• Nothing annotated yet
• Plan: LLM pre-annotation (GPT-4 via spacy-llm) → expert correction in Prodigy
• Set aside ~100 abstracts as held-out eval set before any annotation begins
• Target: ~300–400 annotated abstracts for NER training

Annotation Plan

Schema

• DRUG — drug/substance mention spans (generic, not role-specific)
• ADVERSE_REACTION — adverse reaction term spans (generic)
• Relations: drug-reaction pairs (separate pass after NER is stable)

Schema principles

• Do NOT use composite labels like "DRUG_CAUSING_ADR" — couple NER with role assignment is a mistake
• Normalization to DrugBank / MedDRA is business logic downstream, not part of the annotation schema

Workflow

1. Pilot: 50–100 abstracts, domain expert annotates DRUG spans only (correction of BC5CDR predictions)
2. Evaluate BC5CDR baseline on same set — decide how much fine-tuning is needed
3. ADVERSE_REACTION pass (single-label correction mode)
4. Relation pass: link drug-reaction pairs within sentences
5. Train first model after pilot, run training curve, validate learnability before scaling

Team

• 1–2 domain experts (clinicians / pharma researchers) — ideal small expert team
• Prodigy on Ellf cluster for annotation

Evaluation Strategy

• Hold out ~100 abstracts as a fixed test set before training starts — never train on these
• Evaluate NER with token-level and span-level F1, split by entity type
• Rule-of-10: distinguish 85% vs 90% accuracy needs ~1000 span-level eval examples — plan accordingly
• Relation evaluation: precision/recall on drug-ADR pairs (exact match: both spans + link correct)
• Run training curves after pilot to decide whether more annotation helps or schema needs revision
• Baseline: evaluate scispaCy BC5CDR out-of-the-box before annotating anything

Roadmap

Phase 0 — Text extraction (prerequisite)

• Extract clean abstract text from PDFs
• Tools: Docling, pdfplumber, or spacy-layout
• Output: plain text files per abstract, ready for annotation

Phase 1 — Baseline evaluation

• Run scispaCy en_ner_bc5cdr_md on 50 sample abstracts
• Measure what it gets right on DRUG and ADVERSE_REACTION before committing annotation budget

Phase 2 — NER annotation + model

• LLM pre-annotation → expert correction in Prodigy (ner.correct)
• Single-label passes: DRUG first, ADVERSE_REACTION second
• Train after pilot, run training curve, scale only if curve is still rising

Phase 3 — Relation extraction

• Start with rule-based co-occurrence + negation detection as baseline
• Escalate to NLI reframe ("drug X caused reaction Y" → entailed/neutral/contradicted) if needed

Phase 4 — Normalization (business logic)

• Map DRUG spans to DrugBank / RxNorm
• Map ADVERSE_REACTION spans to MedDRA
• This is a lookup layer, not a model — keep it separate from NLP components

Next Steps

1. Solve text extraction (Phase 0) — this is a blocker for everything else
2. Evaluate scispaCy BC5CDR on a sample before annotating
3. Set up Prodigy annotation task on Ellf with LLM-assisted pre-annotation