Senior AI Agent Architect & Production Evaluator

People in AI; New York (NY); Presencial; USD 225.000 - 275.000; Jornada completa

Descripción de la vacante

People in AI is seeking a Senior Applied AI Engineer in New York City to shape agent architecture, strengthen evaluation infrastructure, and ensure production reliability as usage grows. You’ll own how agents gather context, use tools, and manage multi-step tasks, while building the evaluation and monitoring systems that guide release decisions.

This is a hands-on, high-impact role on a small onsite team. As a senior technical voice, you’ll influence design choices, drive backend improvements,

Formación

• Substantial software engineering experience, including strong production backend ownership.
• Excellent Python skills and strong debugging, testing, and system design fundamentals.
• Demonstrable experience architecting and shipping LLM-powered agents into production.
• Hands-on experience building evaluation datasets, graders, test harnesses, or regression suites.
• A strong understanding of tool calling, orchestration, retrieval, context management, and failure handling.
• Experience balancing quality, speed, and cost in live AI systems.
• Evidence of Staff-level technical leadership while remaining active in code.
• The ability to take ownership, work through ambiguity, and collaborate closely with a small onsite team.

Responsabilidades

• Architect production agents: Design tool use, orchestration, context management, state handling, and guardrails.
• Own evaluation infrastructure: Build representative datasets, automated graders, evaluation harnesses, and regression tests.
• Investigate failures: Trace incorrect outputs back to their causes and turn production issues into lasting improvements.
• Build observability: Track agent behavior, tool calls, task completion, latency, and inference costs.
• Guide model selection: Compare models and approaches using evidence from realistic tasks.
• Strengthen backend systems: Build the services, integrations, and data flows that support reliable agent execution.
• Set technical standards: Help the team make strong decisions around architecture, testing, deployment, and monitoring.

Conocimientos

Production backend

Python

LLM-powered agents

Evaluation datasets

Tool calling & orchestration

Context management

Failure handling

Staff leadership

Herramientas

Test harnesses

Descripción del empleo

People in AI is seeking a Senior Applied AI Engineer in New York City to shape agent architecture, strengthen evaluation infrastructure, and ensure production reliability as usage grows. You’ll own how agents gather context, use tools, and manage multi-step tasks, while building the evaluation and monitoring systems that guide release decisions.

This is a hands-on, high-impact role on a small onsite team. As a senior technical voice, you’ll influence design choices, drive backend improvements,