Principal Data Engineer

ESPN; New York (NY); On-site; USD 180,000 - 241,000; Full time

Job summary

ESPN Product & Technology is seeking a Principal Data Engineer for Marketing and Consumer Data. You will lead identity and device data initiatives, defining architecture, standards, and a roadmap for a petabyte-scale data platform across Databricks and AWS.

You will mentor senior engineers while coordinating with product and privacy teams. You will drive complex cross-system programs, ensure data governance, and build scalable pipelines, leveraging Spark, Hadoop, and modern cloud technologies to

Qualifications

• 10+ years of big data engineering experience building large pipelines.
• 5+ years hands-on performance tuning of distributed systems.
• Experience with Spark, Presto, Hive or similar tools.
• Experience with Snowflake, Redshift, BigQuery or Databricks.
• Knowledge of GDPR/CCPA and data governance practices.
• Proficient in Python, Scala or Java and AWS fundamentals.

Responsibilities

• Define technical strategy for Identity & Device data products and modernization roadmap.
• Lead end-to-end delivery across Identity360 and related datasets.
• Act as SME for identity data, including data modeling and batch/real-time processing.
• Set engineering standards for CI/CD, observability and data governance.
• Mentor engineers through design reviews and coaching.
• Collaborate with Product, QA, Privacy/Legal and other data squads.

Skills

Python

Scala

Java

AWS

S3

Lambda

IAM

Hadoop

Databricks

Spark

Presto

Hive

Snowflake

Redshift

BigQuery

Airflow

CI/CD

Terraform

SCD

Data Modeling

Data Warehousing

Education

Bachelor’s degree in Computer Science, Information Systems, Software, Electrical or Electronics Engineering

Tools

Airflow

Terraform

Job description

Job Requirements

Technology is at the heart of Disney’s past, present, and future. Disney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more - all working to build and advance the technological backbone for Disney’s media business globally.

The team marries technology with creativity to build world-class products, enhance storytelling, and drive velocity, innovation, and scalability for our businesses. We are Storytellers and Innovators. Creators and Builders. Entertainers and Engineers. We work with every part of The Walt Disney Company’s media portfolio to advance the technological foundation and consumer media touch points serving millions of people around the world.

• Building the future of Disney’s media: Our Technologists are designing and building the products and platforms that will power our media, advertising, and distribution businesses for years to come.
• Reach, Scale & Impact: More than ever, Disney’s technology and products serve as a signature doorway for fans' connections with the company’s brands and stories. Disney+. Hulu. ESPN. ABC. ABC News…and many more. These products and brands - and the unmatched stories, storytellers, and events they carry - matter to millions of people globally.
• Innovation: We develop and implement groundbreaking products and techniques that shape industry norms, and solve complex and distinctive technical problems.

Ad Platforms is responsible for Disney’s industry-leading ad technology and products - driving advertising performance, innovation, and value in Disney’s sports, news, and entertainment content, across all media platforms.

Job Summary

As Principal Data Engineer for Marketing and Consumer Data, you will serve as the senior technical leader for Identity and Device data products - including Identity360 and related consumer identifier datasets - that underpin marketing measurement, activation, and analytics across Disney Entertainment & ESPN. You will define technical direction, drive complex cross-system initiatives, establish engineering standards, and mentor leads and senior engineers, while partnering with onshore squad leaders, product, and enterprise architecture to deliver reliable batch and near-real-time data at petabyte scale.

Responsibilities And Duties Of The Role

• Define and drive technical strategy for Identity & Device data products within Marketing and Consumer Data - including pipeline architecture, platform patterns, and modernization roadmap in partnership with onshore squad leaders and enterprise architecture.
• Lead end-to-end delivery of complex, high-impact initiatives across Identity360, device/consumer identifier datasets, and dependent Marketing data products - setting technical approach, sequencing, and quality bar for other engineers.
• Serve as principal SME and escalation point for identity foundational data - data modeling, batch and near-real-time processing, performance at scale, and production reliability - including participation in on-call rotation and incident leadership for critical pipelines.
• Establish and uphold engineering standards for the Identity & Device team and Marketing Data Engineering - code quality, CI/CD, observability, data governance (GDPR, RTBF, CCPA), and reusable platform patterns across Databricks, AWS, and orchestration tooling.
• Mentor and develop engineers through design reviews, pairing, and coaching - building depth in big-data engineering, identity domain knowledge, and operational excellence.
• Partner cross-functionally with Product, QA, Operations, Privacy/Legal, and peer Marketing data squads (Data Collection, Activation, Acquisition) to align requirements, resolve cross-domain dependencies, and represent Bangalore technical decisions to stakeholders.

Required Education, Experience/Skills/Training

Basic Qualifications

• 10+ years of related big data engineering experience modeling and developing large data pipelines and experience with Python, Scala, or Java; strong AWS fundamentals (S3, Lambda, IAM) preferred.
• 5+ years hands-on experience including performance tuning at scale with distributed systems such as Hadoop or Databricks compute using tools/libraries such as Spark, Presto, Hive to query and process large datasets at the petabyte level.
• Experience with at least one major MPP or cloud database technology (Snowflake, Redshift, Big Query, Databricks)
• Solid understanding of Data Modeling techniques and Data Warehousing standard methodologies and practices
• Operational experience in multi-system/services production environment as senior data engineer and point-of-contact / SME
• Problem solving skills with strong attention to detail and excellent analytical and communication skills
• Solid experience with data integration and orchestration toolsets (e.g. Airflow), CI/CD
• Demonstrated ability with cloud infrastructure technologies, including Terraform.
• Strong understanding of slowly changing dimensions (SCD tables) and how to build scalable and performant datasets.

Required Education

• Bachelor’s degree in computer science, Information Systems, Software, Electrical or Electronics Engineering, or comparable field of study, and/or equivalent work experience

Disclaimer

PAY TRANSPARENCY

The hiring range for this position in SFO, CA is $187,900 - $252,000 per year and New York is $179,700 - $241,000 per year. The base pay actually offered will take into account internal equity and also may vary depending on the candidate’s geographic region, job-related knowledge, skills, and experience among other factors. A bonus and/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits, dependent on the level and position offered.

This position is with Disney Entertainment & Sports LLC, which is part of a business we call Disney Entertainment and ESPN Product & Technology.

Disney Entertainment & Sports LLC is an equal opportunity employer. Applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Disney champions a business environment where ideas and decisions from all people help us grow, innovate, create the best stories and be relevant in a constantly evolving world.

DISABILITY ACCOMMODATION FOR EMPLOYMENT APPLICATIONS

The Walt Disney Company and its Affiliated Companies are Equal Employment Opportunity employers and welcome all job seekers including individuals with disabilities and veterans with disabilities. If you have a disability and believe you need a reasonable accommodation in order to search for a job opening or apply for a position, visit the Disney candidate disability accommodations FAQs. We will only respond to those requests that are related to the accessibility of the online application system due to a disability.

Having technical issues? View the FAQs for help.