AI Engineering Lead: Reliable Infra & Evaluation Platform

Jobtailor; New York (NY); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Hybrid schedule

In-office 3 days/week (MWF) in NYC

Job summary

Jobtailor in New York seeks an experienced engineering leader to lead and grow an AI-enabled infrastructure team. You will own technical architecture for workflow agents, drive reliability, and partner with product to set release gates.

You will build evaluation infrastructure, scale internal tooling, and hire engineers, reporting to the VP of Engineering. This hybrid NYC role emphasizes delivery, measurable improvements, and strong leadership across cross-functional teams.

Qualifications

• 5+ years in software engineering, including 1+ year in engineering leadership.
• Direct experience building or shipping a production AI/agentic system and owning its infrastructure.
• Hands-on familiarity with evaluation infrastructure: golden data sets, LLM-as-judge, human annotation, or online monitoring.
• Strong system design and architecture fundamentals.
• Ability to lead through ambiguity and turn unclear, fast-moving problems into structured execution.
• Deep technical fluency in code review, architecture review, and model/system tradeoffs.
• Strong communication skills across technical and non-technical stakeholders, especially product.
• High standards for engineering quality, reliability, and performance.
• Bonus: experience in workflow-heavy, enterprise SaaS, or infrastructure products.
• Bonus: exposure to complex regulated environments.
• Willingness to work a hybrid schedule with 3 in-office days per week (MWF) in New York City
• Authorization to work in the United States or ability to address U.S. sponsorship requirements

Responsibilities

• Lead and grow an engineering team building and hardening AI infrastructure.
• Build engineering culture through clear expectations, 1:1s, feedback loops, and career development.
• Own technical architecture for workflow agents, including orchestration, model serving, latency, cost per task, and failure handling.
• Partner with the Product Manager, AI to define reliability standards and release gates.
• Build and scale evaluation infrastructure, including golden data sets, LLM-as-judge pipelines, human annotation tooling, and online monitoring.
• Drive internal tooling for testing, error analysis, and hill-climbing agentic work.
• Improve reliability, observability, and engineering efficiency across the AI stack.
• Hire and develop engineers with ML/AI systems depth and future team leaders.
• Own the roadmap from technical direction and execution planning through delivery and measurable production accuracy and reliability gains.
• Report directly to the VP of Engineering and partner closely with the Product Manager, AI

Skills

Engineering Leadership

AI/Agentic System Infrastructure

Evaluation Infrastructure

System Design

Architecture Review

Reliability Standards

Performance Optimization

Team Development

Tools

Golden Data Sets

LLM-as-Judge Pipelines

Human Annotation Tooling

Online Monitoring

Internal Tooling

Model Serving

Job description

Jobtailor in New York seeks an experienced engineering leader to lead and grow an AI-enabled infrastructure team. You will own technical architecture for workflow agents, drive reliability, and partner with product to set release gates.

You will build evaluation infrastructure, scale internal tooling, and hire engineers, reporting to the VP of Engineering. This hybrid NYC role emphasizes delivery, measurable improvements, and strong leadership across cross-functional teams.