AI Agent Engineer, Evaluation & Production Systems

People in AI; New York (NY); On-site; USD 200,000 - 250,000; Full time

Job summary

People in AI is hiring an Applied AI Engineer for Agents & Evaluation in New York City, onsite. You will design, build, and iterate on production AI agents, with emphasis on context management, orchestration, prompts, and guardrails to ensure reliability.

You’ll own the evaluation infrastructure, partner with fraud investigators, and build datasets, dashboards, and testing suites to measure and improve agent quality in production.

Qualifications

• 6+ years building production software with backend focus.
• Excellent Python skills and production reliability.
• Hands-on experience shipping LLM-powered agents into production.
• A clear understanding of why production agent systems succeed or fail beyond the demo stage.
• Experience building evaluation datasets, graders, regression frameworks, or evaluation harnesses and using the results to drive product or engineering decisions.
• Strong understanding of tool-using agents, orchestration patterns, and emerging agent protocols and frameworks.
• Experience balancing rapid experimentation with production hardening.

Responsibilities

• Design, build, and iterate on production AI agents, including tools, context management, orchestration, prompts, and guardrails.
• Own the evaluation infrastructure used to measure agent quality before changes reach production.
• Partner directly with fraud investigators to define what a high-quality investigation looks like and translate that into evaluation criteria.
• Build golden datasets, LLM-as-judge systems, rule-based graders, regression suites, and other evaluation harnesses.
• Develop dashboards that make agent quality and failure modes understandable to both technical and non-technical stakeholders.
• Instrument production agents across traces, tool calls, latency, cost, and model behavior.
• Turn real production failures into new test cases and continuously improve the evaluation suite.
• Evaluate new models and providers end-to-end and own production rollouts when the team changes its underlying model stack.
• Partner closely with backend and ML engineers on the systems and data pipelines that agents rely on.
• Build for a high-stakes environment where false findings matter and reliability is a core product requirement.

Skills

Python

LLM Agents

Production software

Testing & reliability

Agent orchestration

Evaluation datasets

Stakeholder collaboration

Job description

People in AI is hiring an Applied AI Engineer for Agents & Evaluation in New York City, onsite. You will design, build, and iterate on production AI agents, with emphasis on context management, orchestration, prompts, and guardrails to ensure reliability.

You’ll own the evaluation infrastructure, partner with fraud investigators, and build datasets, dashboards, and testing suites to measure and improve agent quality in production.