Software Engineer, AI for Incident Response (Technical Leadership)

Meta

New York (NY)

On-site

USD 219,000 - 301,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Meta is seeking a Software Engineer, AI for Incident Response (Technical Leadership) to direct the technical vision and execution for AI-powered incident investigation and mitigation at scale. You will design, implement, and ship core features in a production agent system while mentoring senior engineers across teams.

The role emphasizes reliability, security, latency, and cost, with a path toward self-healing infrastructure at Meta scale.

Qualifications

  • 10+ years of software engineering experience in distributed infra or production AI systems.
  • Experience setting technical direction and leading complex engineering across multiple teams.
  • Hands-on experience coding, debugging, and shipping complex production systems.
  • Experience building systems for always-on production workloads.

Responsibilities

  • Define and drive the technical vision, architecture, and execution for AI-powered incident investigation and mitigation.
  • Personally design, code, evaluate, and ship major capabilities in a production agentic system.
  • Improve root-cause accuracy through better reasoning, skills, memory, and tooling.
  • Build offline evaluation, replay, and online experimentation for continuous improvement of agent behavior.
  • Create a transparent multi-turn experience for collaborative investigations with responders.

Skills

10+ years software engineering
Technical leadership across teams
Production AI systems experience
Hands-on design and shipping
Telemetry and incident analysis

Tools

C++
Java
Python
Rust

Job description

As AI agents increasingly write and operate infrastructure, reliability systems must be able to reason and act at machine speed. In this role, you will set the technical direction and drive execution for an AI agent that autonomously gathers context, investigates incidents, identifies likely root causes, builds a safe mitigation plan, and executes it - with humans supervising and able to intervene at any point.The immediate challenge is building an agent that is both highly accurate and broadly trusted. Experienced incident responders need a transparent collaborator that helps them investigate, continuously surfaces relevant evidence, and keeps them in control. Engineers who encounter incidents only occasionally need an agent that leads the investigation and guides them to a safe outcome.You will lead two tightly connected areas: - Agent quality - skills, tools, context, memory, offline evaluation, backtesting, online experimentation, and learning from production outcomes. - Responder experience - a multi-turn interface that gives responders visibility into the agent's work and brings together the operational data they need to make decisions.This is a deeply hands-on technical leadership role at the intersection of large-scale distributed systems, incident response, applied AI, and product engineering. You will personally design and ship critical parts of the system while leading senior engineers across team and organizational boundaries. The long-term opportunity is to help create self-healing infrastructure that resolves production incidents safely at Meta scale.

Software Engineer, AI for Incident Response (Technical Leadership) Responsibilities
  • Define and drive the technical vision, architecture, and execution roadmap for AI-powered incident investigation and mitigation
  • Personally design, code, evaluate, and ship major capabilities in a production agentic system
  • Improve timely root-cause accuracy through better reasoning, skills, tool use, context, memory, planning, and domain knowledge
  • Build the offline evaluation, historical replay, and online experimentation systems needed to measure changes and continuously improve agent behavior
  • Create a transparent, multi-turn experience that supports collaborative investigation with expert responders and agent-led guidance for less frequent responders
  • Enable the agent to reason across telemetry, code, configuration, deployments, service dependencies, operational knowledge, and incident history
  • Build closed-loop learning systems that turn production outcomes and user feedback into evaluations, experiments, and product improvements
  • Advance the system toward progressive autonomy with explicit authorization, validation, auditability, rollback, and human oversight
  • Build an extensible platform that combines shared capabilities with domain-specific skills, tools, and agents contributed by infrastructure and product teams
  • Partner closely with on-call engineers, reliability experts, product groups, data science, and product management to drive accuracy, adoption, and realized operational impact
  • Ensure the system meets a high bar for reliability, latency, scale, security, privacy, and cost
  • Influence technical strategy across organizations and mentor senior engineers working across distributed systems and applied AI
Minimum Qualifications
  • 10+ years of software engineering experience, including experience building and operating large-scale distributed, infrastructure, or production AI systems
  • Experience setting technical direction and leading complex engineering efforts across multiple teams or organizational boundaries
  • Experience applying AI to production problems at scale, including evaluation of system quality and failure modes
  • Recent hands-on experience designing, coding, debugging, and shipping complex production systems
  • Experience building systems that support critical, always-on production workloads
  • Experience diagnosing complex production systems using telemetry, code, configuration, deployments, and dependency information
  • Experience coding in languages such as C++, Java, Python, Rust, or equivalent
  • Experience influencing senior engineers and leaders without direct organizational authority
Preferred Qualifications
  • Experience improving agent quality through context engineering, skills, memory, evaluation, automated experimentation, hill climbing, fine-tuning, or reinforcement learning
  • Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
  • Experience building multi-turn, human-agent experiences for complex or high-stakes workflows
  • Experience building production AI agents that use tools and reason across telemetry, code, configuration, deployments, and operational knowledge
  • Track record of taking an ambitious technical product from early adoption to trusted, broad production use
  • Experience designing autonomous or semi-autonomous production actions with identity, authorization, independent validation, auditability, rollback, and escalation
  • Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
  • Experience with incident management, SRE, observability, on-call operations, or automated remediation at significant scale
  • Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
  • Experience building extensible platforms that allow domain teams to contribute specialized capabilities while preserving a coherent user experience
About Meta

Meta builds technologies that help people connect, find communities, and grow businesses. When Facebook launched in 2004, it changed the way people connect. Apps like Messenger, Instagram and WhatsApp further empowered billions around the world. Now, Meta is moving beyond 2D screens toward immersive experiences like augmented and virtual reality to help build the next evolution in social technology. People who choose to build their careers by building with us at Meta help shape a future that will take us beyond what digital connection makes possible today—beyond the constraints of screens, the limits of distance, and even the rules of physics.

Meta is proud to be an Equal Employment Opportunity and Affiliated Action employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics. We also consider qualified applicants with criminal histories, consistent with applicable federal, state and local law. Meta participates in the E-Verify program in certain locations, as required by law. Please note that Meta may leverage artificial intelligence and machine learning technologies in connection with applications for employment.

Meta is committed to providing reasonable accommodations for candidates with disabilities in our recruiting process. If you need any assistance or accommodations due to a disability, please let us know at accommodations-ext@meta.com.

$219,000/year to $301,000/year + bonus + equity + benefits

Individual compensation is determined by skills, qualifications, experience, and location. Compensation details listed in this posting reflect the base hourly rate, monthly rate, or annual salary only, and do not include bonus, equity or sales incentives, if applicable. In addition to base compensation, Meta offers benefits. Learn more about benefits at Meta.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer - Agentic Reliability Infrastructure (Technical Leadership)
Software Engineer - Agentic Reliability Infrastructure (Technical Leadership)

Meta • United States

On-site
USD 271,000 - 347,000
Software Engineer, AI for Incident Response (Technical Leadership)
Software Engineer, AI for Incident Response (Technical Leadership)

Meta • Menlo Park (CA)

On-site
USD 219,000 - 301,000
Bonus
Equity
Benefits
Software Engineer - Agentic Reliability Infrastructure (Technical Leadership)
Software Engineer - Agentic Reliability Infrastructure (Technical Leadership)

Meta • New York (NY)

On-site
USD 271,000 - 347,000
Software Engineer - Agentic Reliability Infrastructure (Technical Leadership)
Software Engineer - Agentic Reliability Infrastructure (Technical Leadership)

Meta • Menlo Park (CA)

Remote
USD 271,000 - 347,000
Equity
Bonus
Benefits
Security Engineer, Applied AI
Security Engineer, Applied AI

Meta • New York (NY)

On-site
USD 154,000 - 217,000
Software Engineer, Infrastructure (Technical Leadership) - Central Security
Software Engineer, Infrastructure (Technical Leadership) - Central Security

Meta • Menlo Park (CA)

On-site
USD 219,000 - 301,000
Bonus
Equity
Benefits
Security Engineer - Applied AI
Security Engineer - Applied AI

Meta • New York (NY)

On-site
USD 184,000 - 257,000
Software Engineering Manager, Systems
Software Engineering Manager, Systems

Whatsapp • Menlo Park (CA)

On-site
USD 219,000 - 301,000
AI Specialist (Technical Leadership) - Product and Applied Research
AI Specialist (Technical Leadership) - Product and Applied Research

Meta • Menlo Park (CA)

On-site
USD 219,000 - 301,000
Software Engineer, Infrastructure
Software Engineer, Infrastructure

Meta • New York (NY)

On-site
USD 183,997 - 257,000
Bonus
Equity
Benefits