Senior Software Engineer, AI

Rozgoo; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Lattice’s AI Engineering team builds systems that power AI across our product and user experiences. This is a high-ownership role where you’ll shape evaluation methodology, agent architecture, and core production systems at scale.

You will design an end-to-end evaluation framework, instrument critical metrics, and oversee data pipelines and observability to improve reliability and performance. Bring 5+ years in production AI/ML, deep LLM experience (prompt engineering, RAG, orchestration), and

Qualifications

• 5+ years of professional software engineering with production AI/ML experience.
• Hands-on experience with LLM-based systems: prompt engineering, RAG pipelines, agent orchestration.
• Strong statistical intuition and ability to design experiments.
• Proven ability to build and operate agentic AI systems in production.
• Production-grade Python engineering: clean, maintainable, testable code.

Responsibilities

• Design and ship an end-to-end AI evaluation framework covering offline evals, tracing, and human-in-the-loop feedback.
• Define metrics like task completion, hallucination rates, and user impact.
• Build datasets, test harnesses, and automated scoring pipelines.
• Architect reusable agent infrastructure and RAG pipelines.
• Collaborate with engineers to guide technical direction on agent quality and evaluation strategy.

Skills

Software engineering

Production AI/ML systems

Statistics

Experiment design

Python

Tools

LangGraph

LangSmith

Pinecone

AWS

TypeScript

Job description

This is Engineering at Lattice

Lattice’s Engineering team is continuously improving both our product and our craft. We build maintainable, performant systems using modern technologies, and we collaborate closely with product and design to deliver exceptional user experiences.

Our AI Engineering team is building the systems that power how AI works across Lattice. We’ve laid the foundations: traces are flowing and evals are running - and we’re now focused on defining how our AI products are measured, improved, and trusted at scale. This is a high-ownership role where you’ll help shape evaluation methodology, agent architecture, and the core systems that determine how AI performs in production.

What You Will Do

Evaluation Infrastructure

• Design and ship a robust, end-to-end AI evaluation framework, covering offline evals, production tracing, and human-in-the-loop feedback loops, connected across all of Lattice’s AI use cases.
• Define and instrument the metrics that actually matter: agent task completion, hallucination rates, response quality, user engagement, and downstream business outcomes.
• Build and maintain evaluation datasets, test harnesses, and automated scoring pipelines to catch regressions before they ship.
• Identify and surface the drivers of agent quality improvement, giving the team clear signals on where to invest.

Agent Architecture & Infrastructure

• Architect and implement reusable agent infrastructure: multi-turn conversation workflows, recommendation services, LLM DAGs, and standardized agent topology patterns using LangGraph.
• Build and scale RAG pipelines and retrieval infrastructure, including vector store management and retrieval quality optimization.
• Make principled build vs. buy decisions across LLM providers, agent frameworks, and evaluation tooling, balancing capability, cost, latency, and vendor risk.
• Contribute to production AI systems with a strong focus on reliability, observability, and performance, not just prototypes.

Technical Leadership & Collaboration

• Own projects end-to-end: scope them, drive them to completion, and bring in the right people at the right time.
• Partner with engineering leads and managers to inform technical direction on agent quality and evaluation strategy you’ll be expected to hold intelligent, substantive conversations about methodology, not just implementation.
• Raise the AI engineering bar across the broader team through code review, documentation, and thoughtful technical debate.

Experience

• 5+ years of professional software engineering experience with significant time spent on production AI/ML systems.
• Deep hands-on experience with LLM-based systems: prompt engineering, RAG pipelines, agent orchestration, evaluation metrics, and model fine-tuning.
• Proven ability to work with data and understand statistics, especially in experiments.
• Proven ability to build and operate agentic AI systems in production: multi-step workflows, multi-agent topologies, and the failure modes that come with them.
• Strong command of AI evaluation: you’ve built eval frameworks before, you know the difference between a good eval and a vanity metric, and you have opinions about it.
• Production-grade Python engineering: clean, maintainable, testable code.

Technical Skills

• LangGraph or comparable agent orchestration frameworks. You’ve built real agent workflows with it, not just tutorials.
• LangSmith or comparable LLM observability tooling for tracing, evaluation, and debugging.
• Reads AI papers & blogs regularly and is a trusted source of AI trends.
• Vector databases (Pinecone or similar) and retrieval system design.
• AWS ecosystem or other cloud infrastructure (ex GCP). Comfortable with lambdas, queues, and cloud-native architecture.
• Familiarity with TypeScript is a plus. Our full-stack engineers use it and cross-pollination is valuable.