ML Benchmark Engineer — Remote Production Evaluations

Raydar; United States; Remote; USD 170,000 - 270,000; Full time

Benefits offered by this job

Competitive equity

Remote within the United States

High autonomy

Job summary

Raydar is hiring a Machine Learning Engineer to lead benchmarks and evaluations, partnering with customers, researchers, and a GM to establish the evaluation foundation for foundation models across domains. The role blends ML evaluation, backend infrastructure, and high-ownership customer delivery.

You will design production benchmarks, own data pipelines and execution environments, and create sandboxed setups for agentic evaluations.

Qualifications

• 4+ years of engineering experience, including hands-on ML model evaluation work.
• Experience deploying end-to-end ML evaluation or benchmark systems to production for enterprise timelines.
• Proficiency with ML evaluation frameworks and benchmark design (e.g., LLM-as-judge).

Responsibilities

• Define, design, and build production benchmarks and evaluations with customers and internal researchers.
• Build and own backend infrastructure, including data pipelines, execution environments, storage, and orchestration.
• Create sandboxed environments for agentic evaluations involving tools, code execution, and multi-step tasks.
• Own the engineering portion of customer engagements from technical scoping through production delivery.
• Identify repeatable evaluation patterns and infrastructure gaps that can become scalable products.
• Move quickly through ambiguity while maintaining strong technical judgment and clear written communication.

Skills

Machine learning evaluation

Backend infrastructure

Data pipelines

Customer-facing engineering

LLM evaluation frameworks

Education

Degree in CS/Physics or related field

Tools

Python

GPU/Compute resources

Job description

Raydar is hiring a Machine Learning Engineer to lead benchmarks and evaluations, partnering with customers, researchers, and a GM to establish the evaluation foundation for foundation models across domains. The role blends ML evaluation, backend infrastructure, and high-ownership customer delivery.

You will design production benchmarks, own data pipelines and execution environments, and create sandboxed setups for agentic evaluations.