Remote Senior ML Systems Engineer: AI Evaluation & Pipelines

Devconnectplatform; Northern (KY); Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Competitive salary and equity

Unlimited PTO

Comprehensive health benefits

401k with employer match

Job summary

Hims & Hers is seeking a Senior Staff Engineer to own the end-to-end evaluation of AI models, including data pipelines, gates, and reporting surfaces for clinical and product teams. You will lead cross-team initiatives, set technical direction, and build scalable infrastructure with a focus on safety and accuracy.

The role emphasizes mentoring, collaboration with ML, data science, and platform teams, and delivering reusable standards for evaluation that impact the whole organization.

Qualifications

• 10+ years of experience in ML infrastructure, data engineering, or evaluation/testing systems.
• Hands-on depth in evaluation systems: designing and calibrating LLM judges/scorers, regression-testing methodologies, and adversarial evaluation.
• Experience with dataset versioning, labeling, and high-throughput data pipelines.

Responsibilities

• Own the evaluation system end-to-end, including metric, judge, and scorer design.
• Architect and scale data pipelines, feature and dataset versioning, and calibration workflows.
• Lead multi-team projects to drive standardized, automated evaluation gates.
• Mentor engineers and share methodology across teams.

Skills

Python

ML infrastructure

Data pipelines

Leadership

Communication

Education

Bachelor's degree in a related field

Tools

Databricks

MLflow

Job description

Hims & Hers is seeking a Senior Staff Engineer to own the end-to-end evaluation of AI models, including data pipelines, gates, and reporting surfaces for clinical and product teams. You will lead cross-team initiatives, set technical direction, and build scalable infrastructure with a focus on safety and accuracy.

The role emphasizes mentoring, collaboration with ML, data science, and platform teams, and delivering reusable standards for evaluation that impact the whole organization.