Remote Senior Backend Engineer - AI Evals & Observability

Zillow; United States; On-site; USD 153,000 - 244,000; Full time

Job summary

Zillow is seeking a collaborative backend engineer with strong agentic AI fundamentals to design, build, and scale evaluation frameworks powering AI experiences across the platform.

You will partner with applied scientists, ML engineers, and product leaders to create reliable, scalable systems. The role emphasizes ownership, growth mindset, and the ability to operate in ambiguity while delivering impact.

Qualifications

• 4+ years backend engineering experience with scalable ML services.
• Experience building platform infrastructure or developer-facing services.
• Hands-on experience with ML Evals & Observability frameworks (Databricks MLflow or modern LL M tools).
• Experience evaluating third‑party AI platforms and make build-vs-buy decisions.
• Fluency across applied science, ML, product, design, and engineering.
• Familiarity with orchestration tools (LangChain, LangGraph, MCP).
• Habit of using AI-assisted development tools in your workflow.
• Strong ownership and a growth mindset in ambiguous environments.

Responsibilities

• Design, build, and scale evaluation frameworks for Zillow’s agentic AI experiences.
• Develop tracing, observability, and quality measurement for production AI agents.
• Create platform infrastructure enabling domain teams to build on the Evals platform.
• Collaborate with scientists and engineers to integrate new evaluation capabilities.
• Evolve metrics for AI quality, reliability, and product impact.
• Stay current with agentic AI paradigms and translate them into platform innovations.
• Support scaling, performance, reliability, and cost management for the evaluation layer.

Skills

Backend engineering

Platform infrastructure

ML evals

Build vs buy decisions

Cross‑functional collaboration

Orchestration tools

AI‑assisted tooling

Ownership mindset

Education

Advanced degree in CS/ML or equivalent

Tools

Databricks MLflow

LangSmith

Braintrust

Promptfoo

LangChain

LangGraph

MCP

Job description

Zillow is seeking a collaborative backend engineer with strong agentic AI fundamentals to design, build, and scale evaluation frameworks powering AI experiences across the platform.

You will partner with applied scientists, ML engineers, and product leaders to create reliable, scalable systems. The role emphasizes ownership, growth mindset, and the ability to operate in ambiguity while delivering impact.