Principal Software Engineer – Observability

Jobtailor; California (MO); On-site; USD 180,000 - 240,000; Full time

Job summary

Jobtailor seeks a senior software engineer to design and operate production-grade AI systems with real-time signals and advanced detection.

You will build autonomous agentic workflows, craft frontier-model-based detection, and develop scalable APIs delivering predictive signals and insights. Strong cloud and CI/CD practices are required, along with mentoring and code reviews.

Qualifications

• Bachelor's degree in CS, Engineering, or equivalent experience.
• 10+ years of software engineering experience.
• Experience building AI-powered or data-driven applications at scale.
• Ability to onboard unfamiliar codebases quickly and modernize with AI.

Responsibilities

• Design and operate production-grade AI systems with real-time signals.
• Build autonomous agentic systems using frontier models for detection and resolution.
• Develop scalable APIs delivering predictive signals and insights.
• Improve reliability, costs, and performance of AI-driven platforms.

Skills

AI/ML engineering

Foundation models

LangChain

LangGraph

Cloud platforms (AWS/Azure/GCP)

CI/CD

Microservices

Prompting & context engineering

API design

Education

Bachelor's degree in computer science, Engineering, or equivalent experience

Tools

Cursor

Claude Code

OpenTelemetry

Databricks

Job description

• Design and operate production-grade intelligent systems using real-time signals and AI-driven detection to improve streaming platform, critical service, and customer experience health
• Build and scale autonomous agentic systems using frontier models for detection, root cause analysis, and resolution
• Design AI harnesses, memory, multi-agent orchestration, prompting, context strategies, retrieval, tool routing, and error recovery
• Develop front-end and back-end systems, data and decisioning pipelines, and scalable APIs delivering predictive signals, explainability, and insights
• Optimize AI costs and performance through prompt/context optimization, caching, batching, and model routing
• Apply software engineering best practices, including tested code, code reviews, and CI/CD, while owning production components
• Modernize unfamiliar codebases with AI and embed intelligence into incident response, release validation, and customer insights workflows
• Set technical direction, conduct design and code reviews, and raise the engineering bar
• Improve reliability, observability, and developer productivity across Disney's large-scale streaming ecosystem, including Disney+, Hulu, and ESPN

Requirements

• Bachelor's degree in computer science, Engineering, or equivalent experience
• 10+ years of software engineering experience
• Experience building AI-powered or data-driven applications, scalable APIs, and production systems at scale
• Ability to ramp quickly on unfamiliar codebases and modernize systems with AI while maintaining quality
• Expertise in model prompting, context engineering, harnessing, orchestration patterns, and autonomous agentic workflows
• Strong AI/ML engineering experience with foundation models and frameworks such as LangChain or LangGraph
• Experience building front-end and back-end systems on AWS, Azure, or GCP and integrating frontier model APIs
• Fluency with AI-assisted development tools such as Cursor and Claude Code
• Experience optimizing AI usage for cost and performance through prompt optimization, caching, and model routing
• Experience with GitHub, Docker, AWS/EKS, cloud-native deployments, and mature CI/CD pipelines
• Strong understanding of API design, microservices architecture, and SDLC workflows
• Ability to work independently, set technical direction, mentor engineers, troubleshoot issues, and explain complex concepts to technical and non-technical stakeholders
• Preferred: experience with observability SaaS and high-volume telemetry data, OpenTelemetry, AWS Bedrock, locally or privately hosted open-source models, PySpark, Pandas, Databricks, Snowflake, prompt evaluation, and foundation-model fine-tuning

Core Competencies

Demonstrates expertise in designing and operating AI-driven systems, optimizing performance and costs, and building scalable APIs. Proficient in software engineering best practices, including CI/CD, while mentoring engineers and enhancing system reliability across large-scale streaming platforms.