Remote Lead AI Quality Engineer - Observability & Cost

Hidden Jobs; United States; Remote; USD 180,000 - 260,000; Full time

Benefits offered by this job

Fully remote, async-first team

Biannual company-wide retreats

Job summary

Hidden Jobs is seeking a hands-on Lead Engineer to own evaluation and observability for production LLM-powered agent systems. This role blends player-coach leadership with deep coding contributions, designing pipelines, scorers, and diagnostics to ensure AI agents work correctly while reducing cost and latency.

You will lead the AI Quality engineering team, run experiments, and collaborate with AI engineering to ship reliable, scalable features in a fully remote environment.

Qualifications

• 7+ years building and shipping production software, including work on LLM-powered agents.
• Deep experience with complex tool-using systems and sub-agents, with measurable results.
• Proficiency in Ruby on Rails and Python, or ability to ramp quickly on both.

Responsibilities

• Design and own evaluation infrastructure, including CI/CD pipelines, scorers, and datasets for AI agents across tool calls and multi-turn conversations.
• Trace quality breakdowns across the agent pipeline and translate findings into prioritized fixes or prototypes.
• Build and scale annotation workflows with human, AI-generated, or simulated data to expand product coverage.
• Run experiments comparing prompts, models, and agent configurations, and decide on model swaps, caching, and routing to reduce cost and latency.
• Set technical direction and day-to-day priorities for the AI Quality engineering team while remaining hands-on in code.
• Partner with the core AI engineering team to ensure product changes have measurable quality improvements.

Skills

LLM-based systems

Hands-on leadership

Ruby on Rails

Python

Eval pipelines

Observability infra

Experimentation

Job description

Hidden Jobs is seeking a hands-on Lead Engineer to own evaluation and observability for production LLM-powered agent systems. This role blends player-coach leadership with deep coding contributions, designing pipelines, scorers, and diagnostics to ensure AI agents work correctly while reducing cost and latency.

You will lead the AI Quality engineering team, run experiments, and collaborate with AI engineering to ship reliable, scalable features in a fully remote environment.