Senior ML Systems Engineer — Evaluation & Pipelines Lead

Aimlroles; Northern (KY); Hybrid; USD 180,000 - 260,000; Full time

Benefits offered by this job

Competitive salary & equity

Unlimited PTO

Comprehensive health benefits

ESPP

401k matching

Offsite team retreats

Job summary

Hims & Hers is seeking a senior technologist to own evaluation systems, data pipelines, and cross-team initiatives. You will define the evaluation direction, design scalable pipelines, and lead multi-team projects improving how AI services are assessed and gated.

You will mentor engineers, drive standard approaches, and collaborate with ML, data science, platform engineering, clinical, and product teams to ensure robust validation and safe deployment.

Qualifications

• 10+ years of experience in ML infrastructure, data engineering, or evaluation/testing systems.
• Hands-on depth in evaluation systems: designing and calibrating LLM judges/scorers, regression-testing methodology.
• Hands-on depth in data pipeline engineering: dataset versioning, labeling/calibration workflows, high-throughput ingestion/transformations.
• Experience building standard approaches used by large teams, not just solving a single problem.
• Experience leading multi-team projects to completion and resolving technical disagreements.
• Mentoring engineers and raising the technical bar across teams.
• Excellent communication and ability to tailor messaging to audiences.
• Strong Python and statistical fluency to design/testing frameworks.

Responsibilities

• Set the technical direction for evaluation systems, including metrics, judge design, and infrastructure.
• Design and scale data pipelines for ingestion, transformation, labeling, and calibration.
• Lead multi-team initiatives to define how to evaluate new AI services and automate gates.

Skills

ML infrastructure

Data engineering

Evaluation/testing systems

Python

LLM evaluation

Data pipelines

Regression testing

Mentoring engineers

Tools

Databricks

MLflow

Unity Catalog

Job description

Hims & Hers is seeking a senior technologist to own evaluation systems, data pipelines, and cross-team initiatives. You will define the evaluation direction, design scalable pipelines, and lead multi-team projects improving how AI services are assessed and gated.

You will mentor engineers, drive standard approaches, and collaborate with ML, data science, platform engineering, clinical, and product teams to ensure robust validation and safe deployment.