Applied AI Engineer, Agents & Evaluation

People in AI; New York (NY); On-site; USD 200,000 - 250,000; Full time

Job summary

People in AI is hiring an Applied AI Engineer for Agents & Evaluation in New York City, onsite. You will design, build, and iterate on production AI agents, with emphasis on context management, orchestration, prompts, and guardrails to ensure reliability.

You’ll own the evaluation infrastructure, partner with fraud investigators, and build datasets, dashboards, and testing suites to measure and improve agent quality in production.

Qualifications

• 6+ years building production software with backend focus.
• Excellent Python skills and production reliability.
• Hands-on experience shipping LLM-powered agents into production.
• A clear understanding of why production agent systems succeed or fail beyond the demo stage.
• Experience building evaluation datasets, graders, regression frameworks, or evaluation harnesses and using the results to drive product or engineering decisions.
• Strong understanding of tool-using agents, orchestration patterns, and emerging agent protocols and frameworks.
• Experience balancing rapid experimentation with production hardening.

Responsibilities

• Design, build, and iterate on production AI agents, including tools, context management, orchestration, prompts, and guardrails.
• Own the evaluation infrastructure used to measure agent quality before changes reach production.
• Partner directly with fraud investigators to define what a high-quality investigation looks like and translate that into evaluation criteria.
• Build golden datasets, LLM-as-judge systems, rule-based graders, regression suites, and other evaluation harnesses.
• Develop dashboards that make agent quality and failure modes understandable to both technical and non-technical stakeholders.
• Instrument production agents across traces, tool calls, latency, cost, and model behavior.
• Turn real production failures into new test cases and continuously improve the evaluation suite.
• Evaluate new models and providers end-to-end and own production rollouts when the team changes its underlying model stack.
• Partner closely with backend and ML engineers on the systems and data pipelines that agents rely on.
• Build for a high-stakes environment where false findings matter and reliability is a core product requirement.

Skills

Python

LLM Agents

Production software

Testing & reliability

Agent orchestration

Evaluation datasets

Stakeholder collaboration

Job description

Applied AI Engineer, Agents & Evaluation

Compensation: $200k–$250k base + equity

Location: New York City — fully onsite, 5 days per week

Employment Type: Full-time

Company: A fast-growing, seed-stage AI startup building claims intelligence technology for the property & casualty insurance industry.

This team is applying AI to one of the most complex and adversarial problems in insurance: investigating potential fraud across large volumes of claims, documents, people, businesses, contractors, and associated data.

With the core backend, document-processing, extraction, and traditional ML infrastructure already in place, the company is now building a new generation of agentic systems on top of that foundation.

They’re hiring an Applied AI Engineer, Agents & Evaluation to own both how these agents behave and how the team proves they are working reliably in production.

What You’ll Do

• Design, build, and iterate on production AI agents, including tools, context management, orchestration, prompts, and guardrails.


• Own the evaluation infrastructure used to measure agent quality before changes reach production.


• Partner directly with fraud investigators to define what a high-quality investigation looks like and translate that into evaluation criteria.


• Build golden datasets, LLM-as-judge systems, rule-based graders, regression suites, and other evaluation harnesses.


• Develop dashboards that make agent quality and failure modes understandable to both technical and non-technical stakeholders.


• Instrument production agents across traces, tool calls, latency, cost, and model behavior.


• Turn real production failures into new test cases and continuously improve the evaluation suite.


• Evaluate new models and providers end-to-end and own production rollouts when the team changes its underlying model stack.


• Partner closely with backend and ML engineers on the systems and data pipelines that agents rely on.


• Build for a high-stakes environment where false findings matter and reliability is a core product requirement.

What You’ll Bring

• 6+ years of experience building production software, ideally with a strong backend engineering foundation.


• Excellent Python skills and strong software engineering fundamentals, including testing and production reliability.


• Hands-on experience shipping LLM-powered agents into production.


• A clear understanding of why production agent systems succeed or fail beyond the demo stage.


• Experience building evaluation datasets, graders, regression frameworks, or evaluation harnesses and using the results to drive product or engineering decisions.


• Strong understanding of tool-using agents, orchestration patterns, and emerging agent protocols and frameworks.


• Experience balancing rapid experimentation with production hardening.


• Evidence of meaningful individual ownership over systems used by real customers or internal users.


• The ability to work closely with domain experts and turn ambiguous real-world workflows into measurable AI systems

Why Join?

This is an opportunity to own the complete applied AI feedback loop: defining what “good” looks like, building the agents that perform the work, measuring their performance, observing them in production, and turning failures into better systems.

You’ll be solving problems where evaluation goes far beyond benchmark scores. The agents operate in complex investigations involving incomplete information, interconnected entities, and potentially adversarial behavior, meaning accuracy and reliability have genuine real-world consequences.