Senior Model Evaluation Analyst - ML/NLP Reporting

Ignite IT; Suitland (MD); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

401(k)

401(k) matching

Dental insurance

Flexible spending account

Health insurance

Life insurance

Paid time off

Professional development assistance

Referral program

Tuition reimbursement

Vision insurance

Job summary

Ignite IT Hub is seeking a senior Model Evaluation Analyst to assess the accuracy and reliability of the U.S. Census Bureau's LLM Autocoder.

You will maintain evaluation datasets, run comparisons, investigate coding errors, and produce evidence for Government review. Working with the contract’s Data Scientist and Government SMEs, you will evaluate enhancements for the American Community Survey (ACS) and expansion to other surveys and fields.

Qualifications

• Bachelor's degree in statistics, data science, mathematics, computer science, or related field.
• 8+ years in quantitative analysis, data quality, or model validation, including 3+ years evaluating ML, NLP, or automated classification systems.
• Strong Python and SQL skills for preparing datasets, automating analyses, calculating metrics, and producing reproducible reports.
• Practical understanding of sampling, class imbalance, reference-label quality, data leakage, confusion matrices, and uncertainty in performance estimates.
• Ability to investigate discrepancies independently, document evidence precisely, and explain findings to technical staff and business or domain experts.

Responsibilities

• Prepare and maintain versioned evaluation datasets, reference labels, and test cases with documented provenance and representative coverage of coding categories.
• Execute evaluation methods established with the Senior Data Scientist and calculate accuracy, joint coding rates, coverage, and results against agreed error thresholds.
• Compare candidate models and coding methods with approved baselines; identify material category-level regressions that aggregate results may conceal.
• Analyze difficult, ambiguous, and underrepresented cases; assemble examples for SME adjudication and track decisions and label corrections.
• Develop repeatable Python or SQL analyses and evaluation checks, preserving the data, code, prompt, model, and configuration versions behind each result.
• Maintain test evidence, review-comment disposition, and traceability from requirements and acceptance criteria to evaluated results and deliverables.
• Prepare clear performance reports, tables, and briefings that explain findings and limitations; support pilot assessments, release reviews, and ongoing monitoring.

Skills

Python

SQL

Statistical analysis

NLP evaluation

Education

Bachelor's degree in statistics, data science, mathematics, computer science, or related field

Tools

Python

SQL

Jupyter

Pandas

Job description

Ignite IT Hub is seeking a senior Model Evaluation Analyst to assess the accuracy and reliability of the U.S. Census Bureau's LLM Autocoder.

You will maintain evaluation datasets, run comparisons, investigate coding errors, and produce evidence for Government review. Working with the contract’s Data Scientist and Government SMEs, you will evaluate enhancements for the American Community Survey (ACS) and expansion to other surveys and fields.