AI Inference Quality & Performance Engineer

Fdetimes

Sunnyvale (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Cerebras Systems is seeking a specialist to own model quality and performance for our inference offerings. You will define what good looks like across our deployed models, building AI-driven systems to measure it at scale and translate signals into practical artifacts for customers and product teams.

You will create AI agents-based eval suites for each model release, mine representative test data, automate release qual, and contribute to benchmarking workflows and data views used by customers.

Qualifications

  • Experience building AI agents. You ship real systems with Claude (or equivalent) as a force multiplier. You've built things that would have been infeasible solo without AI agents in the loop.
  • Strong math/stats background.
  • Comfort with Docker, Git, and the standard automation stack
  • A taste for tooling design. You've shipped something that a non-engineer used without complaining. Bonus if AI helped you ship it.

Responsibilities

  • Design eval suites with AI agents in the loop. For every model release, curate a thoughtful mix of advanced, basic, long-context, and customer-use-case-specific evals. Use Claude to generate, validate, and prune candidate test cases at speed.
  • Build custom evals for target customers by orchestrating AI agents to mine trajectories from their workloads and synthesize representative eval sets.
  • Automate eval execution end-to-end with AI-driven pipelines on top of standard tooling (Docker, Git, CI). The goal is a system that runs itself between releases, not a script you re-run by hand.
  • Build automations to forecast and benchmark model performance on Cerebras for our top customers, including modeling how fast customer-specific workloads will run in production.
  • Build product-quality tooling that synthesizes quality + performance data into a single, easy-to-use view.

Skills

AI agents
Math / statistics
Automation stack
Tooling design

Tools

Docker
Git
EvalScope
lm-eval-harness

Job description

Cerebras Systems is seeking a specialist to own model quality and performance for our inference offerings. You will define what good looks like across our deployed models, building AI-driven systems to measure it at scale and translate signals into practical artifacts for customers and product teams.

You will create AI agents-based eval suites for each model release, mine representative test data, automate release qual, and contribute to benchmarking workflows and data views used by customers.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Engineer, Model Quality and Performance
AI Engineer, Model Quality and Performance

Cerebras • Sunnyvale (CA)

On-site
USD 120,000 - 160,000
AI Inference Performance Engineer
AI Inference Performance Engineer

Cerebras Systems, Inc. • Sunnyvale (CA)

On-site
USD 120,000 - 160,000
Senior AI Inference SDET — Release Quality Engineer
Senior AI Inference SDET — Release Quality Engineer

Cerebras • Sunnyvale (CA)

Hybrid
USD 140,000 - 210,000
Staff AI Inference API Engineer
Staff AI Inference API Engineer

FDE Times • Sunnyvale (CA)

On-site
USD 180,000 - 240,000
Senior SDET: AI Inference Platform & Cloud Automation
Senior SDET: AI Inference Platform & Cloud Automation

Cerebras • Sunnyvale (CA)

On-site
USD 120,000 - 170,000
AI Inference Systems Performance Engineer
AI Inference Systems Performance Engineer

Tensordyne • Sunnyvale (CA)

On-site
USD 140,000 - 210,000
Comprehensive benefits
Flexible spending options
Recognition programs
SDET Tech Lead, AI Inference Core & Release Quality
SDET Tech Lead, AI Inference Core & Release Quality

Precision Labs • Sunnyvale (CA)

Hybrid
USD 180,000 - 240,000
AI Engineer, Model Quality and Performance
AI Engineer, Model Quality and Performance

FDE Times • Sunnyvale (CA)

On-site
USD 180,000 - 240,000
Technical Lead Manager, AI Inference Release & QA
Technical Lead Manager, AI Inference Release & QA

Cerebras • Sunnyvale (CA)

On-site
USD 180,000 - 240,000
AI Developer Relations Engineer
AI Developer Relations Engineer

Cerebras • Sunnyvale (CA)

On-site
USD 140,000 - 190,000