Senior Backend Engineer for AI Evaluation Platform

Zillow; United States; On-site; USD 150,000 - 257,000; Full time

Job summary

Zillow is seeking a collaborative backend engineer with strong Agentic AI fundamentals to design, build, and scale evaluation frameworks for AI experiences. You will create tracing, observability, and quality systems for production agents and build infrastructure that enables multiple teams to leverage the Evals platform.

The role emphasizes partnering with scientists and ML engineers, evaluating new evaluation capabilities, and advancing how Zillow measures AI quality, reliability, and impact.

Qualifications

• 4+ years of backend engineering experience with scalable ML services.
• Experience building platform infrastructure consumed by multiple teams; focus on APIs, reliability.
• Hands-on with ML Evals & Observability frameworks (Databricks MLflow or evolving tools).
• Experience evaluating third-party AI platforms and build-vs-buy decisions for platform infra.
• Fluency across applied science, ML, product, design, and engineering; translate between disciplines.
• Familiar with orchestration tools for LLM apps (LangChain, LangGraph, MCP).
• Ownership mindset; comfortable working in ambiguity and delivering results.

Responsibilities

• Design, build, and scale evaluation frameworks for Zillow’s agentic AI experiences.
• Build tracing, observability, and quality measurement systems for production AI agents.
• Create platform infrastructure enabling domain teams to build on the Evals platform.
• Partner with scientists and ML engineers to integrate new evaluation capabilities.
• Evolve how Zillow measures AI quality, reliability, and impact.
• Stay current with agentic AI paradigms and translate into platform innovation.
• Support scaling, reliability, and cost management for the evaluation layer.
• Apply first-principles thinking to ambiguous problems and iterate quickly.

Skills

Backend engineering

APIs & reliability

ML evals frameworks

Build-vs-buy decisions

Cross-disciplinary communication

LLM tooling

Ownership mindset

Education

Advanced degree in CS/ML

Tools

Databricks MLflow

LangChain

LangGraph

MCP

Braintrust

Promptfoo

Job description

Zillow is seeking a collaborative backend engineer with strong Agentic AI fundamentals to design, build, and scale evaluation frameworks for AI experiences. You will create tracing, observability, and quality systems for production agents and build infrastructure that enables multiple teams to leverage the Evals platform.

The role emphasizes partnering with scientists and ML engineers, evaluating new evaluation capabilities, and advancing how Zillow measures AI quality, reliability, and impact.