Lead Product Engineer – (Agentic AI for Observability)

Allstate; United States; Hybrid; USD 100,000 - 171,000; Full time

Job summary

Allstate is seeking a forward-thinking Lead Product Engineer to drive innovation in its AI-powered Observability platform. The role sits at the intersection of AI engineering, observability, and software development, with a focus on automation and self-healing systems across hybrid and multi-cloud environments.

You will lead the design and development of AI-driven monitoring and automation capabilities, shaping product strategy and driving engineering excellence for enterprise-scale

Qualifications

• 5+ years of engineering experience building software, platforms, or automation solutions, with a strong emphasis on observability and distributed systems.
• 4+ years of hands-on experience with observability platforms (Datadog, Dynatrace, OTEL).
• 3+ years of software development experience using Java (Spring Boot), Python, Node.js, or React.
• 2+ years of hands-on experience building agentic AI systems (autonomous agents, AI workflows, or AI-native applications).
• Proven ability to design and implement AI-driven automation and AIOps solutions.
• Experience integrating LLMs, orchestration frameworks, or AI pipelines into production systems.
• Strong experience with Kubernetes and cloud-native architectures.
• Hands-on experience in hybrid environments (on-prem + cloud) across Linux and Windows.
• Expertise in API development, event-driven systems, and microservices architecture.
• Familiarity with CI/CD pipelines, infrastructure-as-code, and DevOps practices.
• Strong problem-solving skills with a focus on automation and innovation.
• Excellent communication skills to articulate complex technical solutions.
• Passion for emerging technologies, particularly AI in observability and operations.

Responsibilities

• AI & Intelligent Automation Design and build agentic AI solutions for observability, including autonomous agents capable of anomaly detection, root cause analysis, and automated remediation.
• Embed AI/ML capabilities into observability platforms to enable predictive insights, anomaly detection, and intelligent alerting.
• Develop AIOps frameworks that reduce noise, automate workflows, and improve incident response times.
• Leverage LLMs and AI orchestration frameworks to create self-service diagnostics and operational assistants.
• Architect and implement end-to-end observability solutions across logs, metrics, traces, and events.
• Enhance observability platforms (Datadog, Dynatrace, New Relic, AppDynamics, OTEL) with custom integrations, automation, and AI enhancements.
• Build real-time insights and health analytics to ensure system reliability, scalability, and performance.
• Lead development of scalable, reusable platform components and APIs for observability and automation.
• Build developer-first tools and frameworks that simplify instrumentation, monitoring, and diagnostics.
• Apply modern software engineering practices including TDD, CI/CD, microservices, and cloud-native design.
• Develop full-stack solutions (backend services, dashboards, automation tooling) using Java/Spring Boot, Python, Node.js, and React.
• Drive automation-first strategies to eliminate manual operations and improve efficiency.
• Build self-healing systems that automatically detect and remediate issues.
• Lead initiatives in infrastructure-as-code, Observability-as-code and automated deployment pipelines.
• Engineer highly resilient systems by focusing on performance, scalability, and fault tolerance.
• Define and evolve enterprise observability architecture, incorporating AI and automation as core principles.
• Collaborate with architecture, platform, and security teams to align solutions with enterprise standards.
• Champion standardization, reusable patterns, and platform scalability across the organization.
• Act as a technical leader and mentor, guiding teams in AI, observability, and engineering best practices.
• Partner with cross-functional teams to deliver integrated, high-impact platform solutions.
• Communicate technical concepts and strategies effectively to both technical and executive audiences.
• Foster a culture of innovation, automation, and continuous improvement.

Skills

Observability

AI in engineering

Java Spring Boot

Python

Node.js

React

Kubernetes

Hybrid cloud

Automation

LLMs integration

Tools

Datadog

Dynatrace

OTEL

Kubernetes

CI/CD tooling

Job description

At Allstate, great things happen when our people work together to protect families and their belongings from life’s uncertainties. And for more than 90 years, our innovative drive has kept us a step ahead of our customers’ evolving needs. From advocating for seat belts, air bags and graduated driving laws, to being an industry leader in pricing sophistication, telematics, and, more recently, device and identity protection.

Job Description Allstate is seeking a forward-thinking Lead Product Engineer to drive innovation in its AI-powered Observability platform.

This role sits at the intersection of AI engineering, observability, and software development, with a strong focus on automation, intelligent insights, and self-healing systems across hybrid and multi-cloud environments. You will lead the design and development of next-generation observability solutions powered by agentic AI, enabling proactive detection, diagnosis, and remediation of issues. This role emphasizes building scalable platforms, intelligent automation, and developer-centric tooling that enhance reliability, performance, and operational efficiency. As a technical leader, you will shape product strategy, drive engineering excellence, and deliver AI-driven monitoring and automation capabilities that transform how applications and platforms are operated across the enterprise.

Key Responsibilities

• AI & Intelligent Automation Design and build agentic AI solutions for observability, including autonomous agents capable of anomaly detection, root cause analysis, and automated remediation.
• Embed AI/ML capabilities into observability platforms to enable predictive insights, anomaly detection, and intelligent alerting.
• Develop AIOps frameworks that reduce noise, automate workflows, and improve incident response times.
• Leverage LLMs and AI orchestration frameworks to create self-service diagnostics and operational assistants.
• Architect and implement end-to-end observability solutions across logs, metrics, traces, and events.
• Enhance observability platforms (Datadog, Dynatrace, New Relic, AppDynamics, OTEL) with custom integrations, automation, and AI enhancements.
• Build real-time insights and health analytics to ensure system reliability, scalability, and performance.
• Lead development of scalable, reusable platform components and APIs for observability and automation.
• Build developer-first tools and frameworks that simplify instrumentation, monitoring, and diagnostics.
• Apply modern software engineering practices including TDD, CI/CD, microservices, and cloud-native design.
• Develop full-stack solutions (backend services, dashboards, automation tooling) using Java/Spring Boot, Python, Node.js, and React.
• Drive automation-first strategies to eliminate manual operations and improve efficiency.
• Build self-healing systems that automatically detect and remediate issues.
• Lead initiatives in infrastructure-as-code, Observability-as-code and automated deployment pipelines.
• Engineer highly resilient systems by focusing on performance, scalability, and fault tolerance.
• Define and evolve enterprise observability architecture, incorporating AI and automation as core principles.
• Collaborate with architecture, platform, and security teams to align solutions with enterprise standards.
• Champion standardization, reusable patterns, and platform scalability across the organization.
• Act as a technical leader and mentor, guiding teams in AI, observability, and engineering best practices.
• Partner with cross-functional teams to deliver integrated, high-impact platform solutions.
• Communicate technical concepts and strategies effectively to both technical and executive audiences.
• Foster a culture of innovation, automation, and continuous improvement.

Essential Skills

• 5+ years of engineering experience building software, platforms, or automation solutions, with a strong emphasis on observability and distributed systems.
• 4+ years of hands-on experience with observability platforms (Datadog, Dynatrace, OTEL).
• 3+ years of software development experience using Java (Spring Boot), Python, Node.js, or React.
• 2+ years of hands-on experience building agentic AI systems (autonomous agents, AI workflows, or AI-native applications).
• Proven ability to design and implement AI-driven automation and AIOps solutions.
• Experience integrating LLMs, orchestration frameworks, or AI pipelines into production systems.
• Strong experience with Kubernetes and cloud-native architectures.
• Hands-on experience in hybrid environments (on-prem + cloud) across Linux and Windows.
• Expertise in API development, event-driven systems, and microservices architecture.
• Familiarity with CI/CD pipelines, infrastructure-as-code, and DevOps practices.
• Strong problem-solving skills with a focus on automation and innovation.
• Excellent communication skills to articulate complex technical solutions.
• Passion for emerging technologies, particularly AI in observability and operations.

This job does not have supervisory duties.

Compensation

Compensation offered for this role is 100,000.00 - 170,500.00 annually and is based on experience and qualifications. The candidate(s) offered this position will be required to submit to a background investigation.

Allstate generally does not sponsor individuals for employment-based visas for this position.

Effective July 1, 2014, under Indiana House Enrolled Act (HEA) 1242, it is against public policy of the State of Indiana and a discriminatory practice for an employer to discriminate against a prospective employee on the basis of status as a veteran by refusing to employ an applicant on the basis that they are a veteran of the armed forces of the United States, a member of the Indiana National Guard or a reserve component.

For jobs in San Francisco, please click “here” for information regarding the San Francisco Fair Chance Ordinance. For jobs in Los Angeles, please click “here” for information regarding the Los Angeles Fair Chance Initiative for Hiring Ordinance. To view the “EEO Know Your Rights” poster click “here”. This poster provides information concerning the laws and procedures for filing complaints of violations of the laws with the Office of Federal Contract Compliance Programs. To view the FMLA poster, click “here”. This poster summarizing the major provisions of the Family and Medical Leave Act (FMLA) and telling employees how to file a complaint.

This policy applies to all aspects of the employment relationship, including, but not limited to, hiring, training, salary administration, promotion, job assignment, benefits, discipline, and separation of employment. It is the Company’s policy to employ the best qualified individuals available for all jobs. Therefore, any discriminatory action taken on account of an employee’s ancestry, age, color, disability, genetic information, gender, gender identity, gender expression, sexual and reproductive health decision, marital status, medical condition, military or veteran status, national origin, race (include traits historically associated with race, including, but not limited to, hair texture and protective hairstyles), religion (including religious dress), sex, or sexual orientation that adversely affects an employee’s terms or conditions of employment is prohibited. This policy applies to all aspects of the employment relationship, including, but not limited to, hiring, training, salary administration, promotion, job assignment, benefits, discipline, and separation of employment.

Allstate provides a comprehensive technology setup, including a laptop, monitors, headset, keyboard, and mouse. Employees eligible to work from home also receive a monthly connectivity reimbursement to help offset internet costs. When working from home, you must have a dedicated, private workspace free from distractions, along with appropriate desk and seating. Reliable internet is required, with minimum speeds of 50 MB download and 5 MB upload.

At Allstate, we work hard to help people live a good life every day. Allstaters are dedicated to serving clients, customers, and communities, which allows employees to find meaning and value in their work. Allstate offers an environment that fosters innovative thinking where you’ll be able to explore your ideas and feel proud of the work you do. Allstate helps protect nearly 16 million households with auto, home, life, and retirement products. We want every professional connected to Allstate to be committed to giving our customers the best and that means finding the best talent. We want you to be our next great addition.