Lead Product Software Engineer - ML Data Systems

Disney Media & Entertainment Distribution; United States; On-site; USD 159,500 - 213,900; Full time

Job summary

Disney Media & Entertainment Distribution’s ESPN division is building a real-time short-form video recommendation system. This Lead Software Engineer role centers on scalable data platforms, APIs, and infrastructure powering personalized experiences at ESPN scale.

You will collaborate with Software Engineering, ML, Product, and Platform teams to design and deliver foundational systems and data platforms, emphasizing low latency, high availability, and cross-team mentorship.

Qualifications

• 7+ years of experience building and maintaining production-grade data pipelines and distributed data processing systems.
• Experience with Spark, Flink, Beam, Kafka Streams, or equivalent.
• Experience designing real-time streaming data pipelines.
• Proficiency with SQL and large-scale analytical datasets.
• Familiarity with cloud data platforms (AWS) and data lake/warehouse components.
• Experience supporting ML workflows (training pipelines, feature engineering).
• Knowledge of data quality frameworks; hands-on with Databricks, Snowflake, Airflow.
• Strong software engineering skills in Python, Java, or Scala.
• Ability to work independently in a fast-paced environment.

Responsibilities

• Design, build, and operate highly scalable software systems and services that support content discovery, personalization, and recommendation experiences.
• Develop and maintain distributed data processing platforms and service architectures that power both online and offline product workflows.
• Build foundational platform capabilities, including feature serving, model inference integration, experimentation infrastructure, and recommendation delivery services.
• Design reliable APIs and service interfaces that enable personalization capabilities across multiple ESPN products and surfaces.
• Lead architecture and technical design efforts for systems that must operate with high availability, low latency, and large-scale traffic demands.
• Partner with Machine Learning, Data Science, Product, and Platform Engineering teams to translate business objectives into scalable software solutions.
• Establish engineering standards, operational best practices, monitoring, observability, and reliability mechanisms across critical systems.
• Drive technical strategy and execution for next-generation personalization platforms and services.
• Mentor engineers and influence engineering practices across teams through technical leadership, design reviews, and architectural guidance.

Skills

Distributed data pipelines

Real-time streaming

SQL

Cloud platforms (AWS)

Python/Java/Scala

Machine Learning workflows

Data quality frameworks

Data orchestration (Airflow)

MLOps familiarity

Education

Bachelor’s or Master’s in CS/Data Engineering

Tools

Spark

Flink

Beam

Kafka Streams

Databricks

Snowflake

Airflow

dbt

Kubernetes

AWS S3 / Data Lakes

Job description

Lead Product Software Engineer – ML Data Systems

Dept : ESPN Product & Technology

Department/Group Overview

Technology is at the heart of Disney’s past, present, and future. Disney Sports News & Entertainment is a global organization of engineers, product developers, designers, technologists, data scientists, and more – all working to build and advance the technological backbone for Disney’s media business globally.

The team marries technology with creativity to build world‑class products, enhance storytelling, and drive velocity, innovation, and scalability for our businesses. We are Storytellers and Innovators. Creators and Builders. Entertainers and Engineers. We work with every part of The Walt Disney Company’s media portfolio to advance the technological foundation and consumer media touch points serving millions of people around the world.

Building the future of ESPN’s media: Our Technologists are designing and building the products and platforms that will power our media, advertising, and distribution businesses for years to come. Reach, Scale & Impact: More than ever, Disney’s technology and products serve as a signature doorway for fans’ connections with the company’s brands and stories. Disney+. Hulu. ESPN. ABC. ABC News…and many more. These products and brands – and the unmatched stories, storytellers, and events they carry – matter to millions of people globally. Innovation: We develop and implement groundbreaking products and techniques that shape industry norms, and solve complex and distinctive technical problems.

Product Engineering is a unified team responsible for the engineering of Disney Entertainment & ESPN digital and streaming products and platforms. This includes product engineering, media engineering, quality assurance, engineering behind personalization, commerce, lifecycle, and identity.

Job Summary

ESPN is building a new real‑time short‑form video recommendation system that will be the foundation of our next‑generation personalization experience. High‑quality data is at the core of this effort. We are seeking a Lead Software Engineer with deep expertise in building scalable distributed systems, platform services, and data‑intensive applications that power personalized user experiences. In this role, you will work closely with Software Engineering, Machine Learning, Product, and Platform teams to design and deliver the foundational systems, APIs, data platforms, and infrastructure that support real‑time personalization and recommendation services at ESPN scale.

Responsibilities and Duties of the Role

• Design, build, and operate highly scalable software systems and services that support content discovery, personalization, and recommendation experiences.
• Develop and maintain distributed data processing platforms and service architectures that power both online and offline product workflows.
• Build foundational platform capabilities, including feature serving, model inference integration, experimentation infrastructure, and recommendation delivery services.
• Design reliable APIs and service interfaces that enable personalization capabilities across multiple ESPN products and surfaces.
• Lead architecture and technical design efforts for systems that must operate with high availability, low latency, and large‑scale traffic demands.
• Partner with Machine Learning, Data Science, Product, and Platform Engineering teams to translate business objectives into scalable software solutions.
• Establish engineering standards, operational best practices, monitoring, observability, and reliability mechanisms across critical systems.
• Drive technical strategy and execution for next‑generation personalization platforms and services.
• Mentor engineers and influence engineering practices across teams through technical leadership, design reviews, and architectural guidance.

Required Education, Experience/Skills/Training

Basic Qualifications

• 7+ years of experience building and maintaining production‑grade data pipelines and distributed data processing systems
• Strong experience with modern data processing frameworks such as Spark, Flink, Beam, Kafka Streams, or equivalent.
• Experience designing and implementing real‑time streaming data pipelines.
• Proficiency with SQL and schema design for large‑scale analytical datasets.
• Familiarity with cloud data platforms (e.g., AWS) and modern data infrastructure components (e.g., data lakes, data warehouses, feature stores).
• Experience supporting ML workflows (model training pipelines, feature engineering, data validation).
• Strong knowledge of data quality frameworks and best practices, with hands‑on experience using Databricks, Snowflake, and Apache Airflow for data pipeline orchestration and validation.
• Solid software engineering skills with experience in Python, Java, Scala, or similar languages.
• Strong problem‑solving skills and ability to work independently in a fast‑paced environment.

Preferred Qualifications

• Prior experience building data infrastructure for personalization, recommendation systems, or other ML‑powered products.
• Familiarity with ML lifecycle tools (MLflow, TFX, Kubeflow) and MLOps best practices.
• Experience implementing data validation, monitoring, and lineage tools (e.g., dbt tests, Snowflake data quality checks) to ensure high data integrity for ML models.
• Knowledge of real‑time ML serving architectures and online feature generation.
• Experience optimizing large‑scale data workflows for latency‑sensitive applications.
• Prior experience operating in 0→1 product development or startup environments.
• Nice to have experience with tools/technologies such as Databricks, Snowflake, Kafka, AWS SQS, Kubernetes, and related cloud‑native data platform components.

Required Education

• Bachelor’s or Master’s in Computer Science, Data Engineering, or a related technical field, or equivalent practical experience.
  The hiring range for this position in CT and CA is $155,700 – $208,700, and in NY is $159,500 – $213,900 per year. The base pay actually offered will take into account internal equity and also may vary depending on the candidate’s geographic region, job‑related knowledge, skills, and experience among other factors. A bonus and/or long‑term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.