Senior LLM Evaluation Infrastructure Engineer

Inception; San Francisco (CA); On-site; USD 200,000 - 350,000; Full time

Benefits offered by this job

Health insurance

Equity

Flexible vacation

Catered meals

PTO

Job summary

Inception seeks seasoned engineers and scientists to build evaluation infrastructure for frontier LLMs. Youll design frameworks to measure model improvements and ensure reliable production performance.

Join a team of AI researchers and engineers in Palo Alto, contributing to scalable evaluation pipelines, metrics, and benchmarking across tasks, with equity and competitive compensation.

Qualifications

• BS/MS/PhD in Computer Science, Machine Learning, Statistics, or a related field (or equivalent experience).
• At least 2 years of experience in ML evaluation, applied ML research, or a related engineering role.
• Experience with version control (Git), containerization (Docker), and cloud services (AWS, GCP, or Azure).
• Understanding of LLM fundamentals (autoregressive generation, instruction tuning, RLHF, in-context learning, decoding strategies).
• Proficiency in Python and ML frameworks such as PyTorch.
• Experience designing and implementing evaluation metrics and benchmarks for generative models.
• Excellent communication skills with the ability to distill complex evaluation results into actionable insights.

Responsibilities

• Build scalable, automated evaluation pipelines that integrate into model training and deployment workflows.
• Design, develop, and maintain robust evaluation frameworks and benchmarks for measuring LLM performance across diverse tasks and domains.
• Conduct rigorous statistical analysis of model outputs to identify failure modes, biases, and performance gaps.
• Partner with product and customer-facing teams to translate real-world use cases into meaningful evaluation criteria.
• Define and implement quantitative metrics that capture model quality, safety, reliability, and regression detection.

Skills

ML evaluation

Python

PyTorch

Git

Docker

Cloud services

Communication

Statistical analysis

Education

BS/MS/PhD in CS/ML or related field

Tools

Docker

Kubernetes

Terraform

Job description

Inception seeks seasoned engineers and scientists to build evaluation infrastructure for frontier LLMs. Youll design frameworks to measure model improvements and ensure reliable production performance.

Join a team of AI researchers and engineers in Palo Alto, contributing to scalable evaluation pipelines, metrics, and benchmarking across tasks, with equity and competitive compensation.