AI Engineer, Evaluation

Distyl; San Francisco (CA); Hybrid; USD 150,000 - 250,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

401(k) with commuter benefits

Access to modern AI tools

Job summary

Distyl is seeking an AI Evaluation Engineer to design evaluation systems for AI-driven solutions. Located in San Francisco, this position requires strong Python skills and a commitment to utilizing AI tools to enhance evaluation processes.

The role offers a competitive salary range of $150K – $250K based on experience, with comprehensive benefits including medical coverage and 401(k) plans. You will be working in a hybrid model with 3+ days per week in the office.

Qualifications

• 2+ years of software engineering experience.
• Strong Python engineering skills, clean and maintainable code.
• Experience with Evaluation-Driven Development frameworks.

Responsibilities

• Design and implement evaluation frameworks for AI systems.
• Define metrics and build test cases in Python.
• Develop evaluation pipelines integrating directly into system iteration loops.

Skills

Python engineering skills

Understanding Evaluation-driven Development

Systems-oriented mindset

AI-native working style

Job description

About Distyl

Distyl is an applied AI technology company partnering with the world’s most ambitious institutions to rearchitect critical operations for the frontier of AI. Our customers include the largest companies in telecom, healthcare, insurance, manufacturing, consumer goods, and global social organizations. We research and deploy technologies that power AI‑native operations—both for our partners and for Distyl itself. Our work spans research into self‑constructing systems, the development of the most reliable execution of AI systems, and products that transform mission‑critical workflows. Our technologies affect some of the world’s largest operations—from hundreds of millions of consumer interactions to tens of millions of supply‑chain transactions and millions of patient journeys. Distyl is backed by leading investors including Lightspeed Venture Partners, Khosla Ventures, Coatue, DST Global, and board members of 20+ Fortune 500 companies. We maintain a 100% production deployment success rate for our customers and operate as a profitable AI enterprise.

What We Are Looking For

At Distyl, we build AI systems using Evaluation‑Driven Development—a process where evaluation is the primary mechanism for iterating, improving, and trusting AI behavior in production. AI Evaluation Engineers design and implement the evaluation systems that drive this process. They are hands‑on engineers who write production Python code, build evaluation pipelines, and use structured signals to guide system design, prompt iteration, and deployment decisions for real customer‑facing AI systems.

Key Responsibilities

• Design and implement evaluation frameworks that enable Evaluation‑Driven Development for AI systems deployed in customer environments.
• Define how system quality is measured in each domain, ensuring evaluation signals reflect real user needs, domain constraints, and business objectives.
• Build and maintain golden test cases and regression suites in Python, using both human‑authored and AI‑assisted test generation to capture critical behaviors and edge cases. These test suites are treated as first‑class system components that evolve alongside the AI system itself.
• Develop and maintain evaluation pipelines—offline and online—that integrate directly into system iteration loops. Evaluation results inform prompt design, agent logic, model selection, and release readiness, ensuring that system changes are driven by measurable improvements rather than intuition alone.
• Define, calibrate, and operate LLM‑based graders, aligning automated judgments with expert human assessments. Investigate where evaluation signals diverge from real‑world outcomes and refine grading approaches to maintain signal quality as systems and domains evolve.
• Work closely with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts to ensure evaluation frameworks meaningfully guide system development and deployment in production.

What We Require

• 2+ years of software engineering experience.
• Strong Python engineering skills: write clean, maintainable Python and build evaluation and experimentation pipelines that run in production environments.
• Experience with Evaluation‑Driven or Experiment‑Driven Development: using structured evaluation or experimentation frameworks to drive system iteration, and understand the pitfalls of overfitting to metrics that don’t reflect real outcomes.
• Ability to translate human judgment into code: elicit high‑quality judgments from subject matter experts and encode them into test cases, scoring functions, and graders that scale.
• Systems‑oriented mindset: understand how evaluation interacts with prompts, agents, data, and deployment. Design evaluation systems that support fast iteration while maintaining trust and safety in production.
• AI‑native working style: use AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration.
• Travel: 10‑50% travel depending on the project, role level, and personal interest.

What We Offer

• Base salary range: $150K – $250K, depending on experience, location, and level, plus meaningful equity and a comprehensive benefits package.
• Medical, dental, and vision coverage at 100% for employees and dependents.
• 401(k) with additional perks such as commuter benefits and in‑office lunch.
• Access to state‑of‑the‑art models, generous usage of modern AI tools, and real‑world business problems.
• Ownership of high‑impact projects across top enterprises.
• A mission‑driven, fast‑moving culture that prizes curiosity, pragmatism, and excellence.

Distyl has offices in San Francisco and New York. This role follows a hybrid collaboration model with 3+ days per week (Tuesday–Thursday) in‑office.

We believe diverse perspectives make our work stronger and more impactful. We are an equal‑opportunity employer and evaluate all applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, disability, veteran status, or any other legally protected characteristic. We encourage candidates from all backgrounds to apply.