Lead Principal: AI-Driven Production Reliability Engineer

Meta; Helena (MT); On-site; USD 271,000 - 347,000; Full time

Job summary

Meta is seeking an experienced Software Engineer to build the next generation of AI systems for production reliability. This role sits at the intersection of large-scale distributed systems, incident response, and applied AI.

You will lead the development of an AI agent that can autonomously investigate production incidents, identify likely root causes, create safe mitigation plans, and execute them while humans supervise and can intervene.

Qualifications

• Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.
• 12+ years of software engineering experience building and operating large-scale distributed or infrastructure systems.
• Experience setting technical direction and leading complex, multi-year engineering efforts across organizational boundaries.
• Experience applying AI or machine learning systems to production problems.
• Demonstrated experience moving from technical concept to production deployment and measurable impact.

Responsibilities

• Define the technical vision and architecture for agentic reliability systems across Meta.
• Personally design, code, and ship production agentic systems for complex infrastructure problems.
• Lead the development of an AI agent for production incident investigation and mitigation, advancing it toward accurate, trusted, and safely supervised autonomous action.
• Develop major improvements in agent reasoning, context handling, planning, evaluation, learning, and safe execution.
• Explore techniques including automated hill climbing, fine-tuning, reinforcement learning, model routing, distillation, pruning, and inference optimization, selecting the simplest approach that produces measurable gains.
• Identify new high-value applications of AI across incident prevention, detection, mitigation, observability, and infrastructure operations.
• Move rapidly from ambiguous problems to prototypes, validate them against real production workloads, and develop successful approaches into reliable systems at scale.
• Build evaluation and experimentation systems that connect agent quality to outcomes such as investigation accuracy, successful mitigation, incident duration, and reduced operational work.
• Build closed-loop improvement systems that turn production outcomes into evaluations, experiments, and better agent behavior.
• Establish architectures and guardrails for production actions, including authorization, independent validation, auditability, rollback, and human oversight.
• Partner with Infrastructure, AI, Product, and Reliability leaders to integrate agentic capabilities into Meta’s production ecosystem.
• Influence technical strategy across organizations and mentor other engineers working on distributed systems and applied AI.

Skills

C++

Java

Python

Rust

Education

Bachelor's degree in Computer Science or related field

Job description

Meta is seeking an experienced Software Engineer to build the next generation of AI systems for production reliability. This role sits at the intersection of large-scale distributed systems, incident response, and applied AI.

You will lead the development of an AI agent that can autonomously investigate production incidents, identify likely root causes, create safe mitigation plans, and execute them while humans supervise and can intervene.