Senior AI Engineer — Evaluation & Agent Infra

Rozgoo; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Lattice’s AI Engineering team builds systems that power AI across our product and user experiences. This is a high-ownership role where you’ll shape evaluation methodology, agent architecture, and core production systems at scale.

You will design an end-to-end evaluation framework, instrument critical metrics, and oversee data pipelines and observability to improve reliability and performance. Bring 5+ years in production AI/ML, deep LLM experience (prompt engineering, RAG, orchestration), and

Qualifications

• 5+ years of professional software engineering with production AI/ML experience.
• Hands-on experience with LLM-based systems: prompt engineering, RAG pipelines, agent orchestration.
• Strong statistical intuition and ability to design experiments.
• Proven ability to build and operate agentic AI systems in production.
• Production-grade Python engineering: clean, maintainable, testable code.

Responsibilities

• Design and ship an end-to-end AI evaluation framework covering offline evals, tracing, and human-in-the-loop feedback.
• Define metrics like task completion, hallucination rates, and user impact.
• Build datasets, test harnesses, and automated scoring pipelines.
• Architect reusable agent infrastructure and RAG pipelines.
• Collaborate with engineers to guide technical direction on agent quality and evaluation strategy.

Skills

Software engineering

Production AI/ML systems

Statistics

Experiment design

Python

Tools

LangGraph

LangSmith

Pinecone

AWS

TypeScript

Job description

Lattice’s AI Engineering team builds systems that power AI across our product and user experiences. This is a high-ownership role where you’ll shape evaluation methodology, agent architecture, and core production systems at scale.

You will design an end-to-end evaluation framework, instrument critical metrics, and oversee data pipelines and observability to improve reliability and performance. Bring 5+ years in production AI/ML, deep LLM experience (prompt engineering, RAG, orchestration), and