Senior AI Platform Engineer & Quality Lead

Lever, Inc.; Ireland; Remote; EUR 136,000 - 166,000; Full time

Benefits offered by this job

Annual cash compensation: $170,000 USD

Equity with refresh grants

35 days paid time off per year

Fully remote work environment

Global company retreats

Job summary

Lever, Inc. seeks a Lead Engineer, AI Platform in Ireland to build evaluation frameworks, observability tooling, and diagnostics for production AI agents. You will lead hands-on engineering and people management, shaping the AI Quality function.

In a highly remote, asynchronous environment, you will guide experiments, datasets, and model comparisons, while optimizing cost, latency, and user experience across complex AI workflows.

Qualifications

• 7+ years of experience building and shipping production software.
• Experience with complex, tool-using AI systems involving multiple tools, planning, orchestration, or sub-agents.
• Strong ability to demonstrate shipped software and explain how its effectiveness and reliability were measured.
• Experience with Ruby on Rails and/or Python, with the ability to become productive quickly in technologies that may be new to you.
• Experience building evaluation or observability infrastructure for ML/AI systems, including evaluation pipelines, scorers, dashboards, or CI/CD systems for evaluations.
• Familiarity with evaluation frameworks such as Braintrust, LangSmith, or similar tools.
• Experience designing datasets, annotation workflows, or labeling pipelines for machine learning or AI evaluation.
• Ability to learn quickly, experiment extensively, and use empirical results to guide technical decisions.
• Comfortable operating in a fast-paced environment with ambiguity and changing technical requirements.
• Strong technical leadership and people-management capabilities, with the ability to balance team leadership and hands-on engineering.
• Excellent English proficiency in spoken, written, and reading communication, equivalent to CEFR C2 / ILR 5.

Responsibilities

• Design, build, and own evaluation infrastructure, including CI/CD pipelines, scorers, datasets, and systems for assessing AI agents from individual tool calls through complete multi-turn conversations.
• Develop observability and diagnostic capabilities to identify exactly where quality issues occur across planning, execution, tool selection, and complex agent trajectories.
• Investigate failures across sophisticated AI workflows and turn findings into technical prototypes, improvements, or clearly defined priorities for AI engineering teams.
• Build and expand datasets through human annotation, AI-generated examples, and simulated conversations to increase evaluation coverage efficiently.
• Develop structured experimentation frameworks for prompts, models, and agent harnesses, including evaluation of new and open-source models against production baselines.
• Identify opportunities to improve AI system cost and latency through model selection, caching, routing, and other optimization strategies.
• Set the technical direction and manage day-to-day priorities for the AI Quality engineering team while remaining actively involved in hands-on development.
• Partner closely with AI Core engineering teams to ensure changes to AI products can be measured effectively and demonstrably improve quality.
• Establish engineering practices and evaluation approaches that support reliable, efficient, and scalable production AI systems.

Skills

Technical leadership

People management

English proficiency

Hands-on engineering

Fast-paced environment

Empirical decision making

Tools

Ruby on Rails

Python

Braintrust

LangSmith

Job description

Lever, Inc. seeks a Lead Engineer, AI Platform in Ireland to build evaluation frameworks, observability tooling, and diagnostics for production AI agents. You will lead hands-on engineering and people management, shaping the AI Quality function.

In a highly remote, asynchronous environment, you will guide experiments, datasets, and model comparisons, while optimizing cost, latency, and user experience across complex AI workflows.