Senior Applied AI Engineer

People in AI; New York (NY); On-site; USD 225,000 - 275,000; Full time

Job summary

People in AI is seeking a Senior Applied AI Engineer in New York City to shape agent architecture, strengthen evaluation infrastructure, and ensure production reliability as usage grows. You’ll own how agents gather context, use tools, and manage multi-step tasks, while building the evaluation and monitoring systems that guide release decisions.

This is a hands-on, high-impact role on a small onsite team. As a senior technical voice, you’ll influence design choices, drive backend improvements,

Qualifications

• Substantial software engineering experience, including strong production backend ownership.
• Excellent Python skills and strong debugging, testing, and system design fundamentals.
• Demonstrable experience architecting and shipping LLM-powered agents into production.
• Hands-on experience building evaluation datasets, graders, test harnesses, or regression suites.
• A strong understanding of tool calling, orchestration, retrieval, context management, and failure handling.
• Experience balancing quality, speed, and cost in live AI systems.
• Evidence of Staff-level technical leadership while remaining active in code.
• The ability to take ownership, work through ambiguity, and collaborate closely with a small onsite team.

Responsibilities

• Architect production agents: Design tool use, orchestration, context management, state handling, and guardrails.
• Own evaluation infrastructure: Build representative datasets, automated graders, evaluation harnesses, and regression tests.
• Investigate failures: Trace incorrect outputs back to their causes and turn production issues into lasting improvements.
• Build observability: Track agent behavior, tool calls, task completion, latency, and inference costs.
• Guide model selection: Compare models and approaches using evidence from realistic tasks.
• Strengthen backend systems: Build the services, integrations, and data flows that support reliable agent execution.
• Set technical standards: Help the team make strong decisions around architecture, testing, deployment, and monitoring.

Skills

Production backend

Python

LLM-powered agents

Evaluation datasets

Tool calling & orchestration

Context management

Failure handling

Staff leadership

Tools

Test harnesses

Job description

Compensation: $225,000–$275,000 base + equity

Location: New York City | 5 days onsite

Employment Type: Full-time

Join a venture-backed AI startup building production agents for complex decisions where accuracy and trust matter.

Our client is developing intelligent systems that bring together fragmented information, uncover meaningful connections, and help users work through demanding, evidence-heavy workflows.

The company has paying customers and strong commercial momentum. They're now looking for a Senior Applied AI Engineer to shape agent architecture, strengthen evaluation infrastructure, and help the product perform reliably as usage grows.

The Opportunity

This is a hands‑on engineering role at the intersection of LLM agents, backend systems, and evaluation.

You'll own how agents gather context, use tools, move through multi‑step tasks, and recover when something goes wrong. You'll also build the evaluation and monitoring systems that help the team understand performance and make sound release decisions.

As a senior technical voice on a small team, you'll have significant influence over what gets built and how it works in production.

What You'll Do

• Architect production agents: Design tool use, orchestration, context management, state handling, and guardrails.
• Own evaluation infrastructure: Build representative datasets, automated graders, evaluation harnesses, and regression tests.
• Investigate failures: Trace incorrect outputs back to their causes and turn production issues into lasting improvements.
• Build observability: Track agent behavior, tool calls, task completion, latency, and inference costs.
• Guide model selection: Compare models and approaches using evidence from realistic tasks.
• Strengthen backend systems: Build the services, integrations, and data flows that support reliable agent execution.
• Set technical standards: Help the team make strong decisions around architecture, testing, deployment, and monitoring.

What You'll Bring

• Substantial software engineering experience, including strong production backend ownership.
• Excellent Python skills and strong debugging, testing, and system design fundamentals.
• Demonstrable experience architecting and shipping LLM-powered agents into production.
• Hands‑on experience building evaluation datasets, graders, test harnesses, or regression suites.
• A strong understanding of tool calling, orchestration, retrieval, context management, and failure handling.
• Experience balancing quality, speed, and cost in live AI systems.
• Evidence of Staff‑level technical leadership while remaining active in code.
• The ability to take ownership, work through ambiguity, and collaborate closely with a small onsite team.