Senior ML Model Evaluator (LLM) & Insights Lead

Ignite IT, LLC; Suitland, Northern (MD, KY); Hybrid; USD 110,000 - 150,000; Full time

Benefits offered by this job

401(k)

401(k) matching

Dental insurance

Flexible spending account

Health insurance

Life insurance

Paid time off

Professional development assistance

Referral program

Tuition reimbursement

Vision insurance

Job summary

Working with the contract’s Data Scientist and Government SMEs, you will evaluate enhancements for the American Community Survey (ACS) and expansion to other surveys and fields. Responsibilities include developing repeatable Python or SQL analyses, maintaining test evidence, and presenting clear performance reports and briefings.

Qualifications

• Bachelor's degree in statistics, data science, mathematics, computer science, or related field, or equivalent professional experience.
• 8+ years in quantitative analysis, data quality, or model validation, including 3+ years evaluating ML, NLP, or automated classification systems.
• Strong Python and SQL skills for preparing datasets, automating analyses, calculating metrics, and producing reproducible reports.
• Understanding of sampling, class imbalance, reference-label quality, data leakage, confusion matrices, and uncertainty in performance estimates.
• Ability to investigate discrepancies, document evidence precisely, and explain findings to technical staff and domain experts.

Responsibilities

• Prepare and maintain versioned evaluation datasets, reference labels, and test cases with provenance and coverage.
• Execute evaluation methods and calculate accuracy, joint coding rates, coverage, and results against thresholds.
• Compare candidate models and coding methods with baselines; identify regressions that may be concealed in aggregate results.
• Analyze difficult or underrepresented cases and collect examples for SME adjudication.
• Develop repeatable Python or SQL analyses and preserve data, code, prompts, models, and configurations behind results.
• Maintain evidence, review dispositions, and ensure traceability from requirements to results.
• Prepare clear performance reports, tables, and briefings to support pilot assessments and monitoring.

Skills

Python

SQL

Quantitative analysis

Data quality

Model validation

Education

Bachelor's degree in statistics or related field

Tools

Python

Job description

Working with the contract’s Data Scientist and Government SMEs, you will evaluate enhancements for the American Community Survey (ACS) and expansion to other surveys and fields. Responsibilities include developing repeatable Python or SQL analyses, maintaining test evidence, and presenting clear performance reports and briefings.