AI Evaluation Engineer - Reproducible ML Benchmarks Hybrid

SHRM; Alexandria (VA); Hybrid; USD 100,000 - 130,000; Full time

Benefits offered by this job

Health benefits

Retirement plan

Bonuses & incentives

Professional growth

Job summary

SHRM seeks a Senior Machine Learning Engineer, AI Evaluation to design and operate the measurement and engineering infrastructure for AAIR, supporting rigorous evaluation of AI models against professional standards.

You will build scalable evaluation systems, versioned experiments, and data pipelines while partnering with HR experts to translate domain criteria into measurable specs; this role serves multiple AI research workstreams with a focus on reproducibility and auditable findings.

Qualifications

• 7+ years of experience in ML/LLM engineering, applied AI, or research infrastructure.
• Hands-on experience with multi-model LLM applications and infrastructure.
• Experience with AI model evaluation/benchmarking, rubric-based scoring, and inter-rater reliability.
• Cloud-based data/AI infrastructure experience including BigQuery and Vertex AI.

Responsibilities

• Design, build, and maintain scalable engineering infrastructure for evaluations across multiple AI/LLM families.
• Develop a provider-agnostic orchestration layer for consistent evaluation across frontier model providers.
• Design and implement rigorous AI evaluation and scoring frameworks, including rubric-based scoring and model-as-judge methods.
• Build systems for reproducible experimentation with versioning, logging, and drift detection.
• Maintain data infrastructure for experiment results, prompts, rubrics, and metadata.
• Collaborate with HR experts to translate standards into measurable evaluation specifications.
• Communicate methodologies and findings clearly to technical and non-technical audiences.

Skills

Python

ML/LLM

AI evaluation

Experiment tracking

Cloud platforms

Communication

Education

Bachelor's in Computer Science/Data Science/ML/Engineering

Master's preferred

Tools

BigQuery

Vertex AI

Looker Studio

Job description

SHRM seeks a Senior Machine Learning Engineer, AI Evaluation to design and operate the measurement and engineering infrastructure for AAIR, supporting rigorous evaluation of AI models against professional standards.

You will build scalable evaluation systems, versioned experiments, and data pipelines while partnering with HR experts to translate domain criteria into measurable specs; this role serves multiple AI research workstreams with a focus on reproducibility and auditable findings.