Senior Data Engineer - Identity & Big Data Pipelines

The Walt Disney Company; San Francisco (CA); On-site; USD 155,000 - 208,000; Full time

Job summary

The Walt Disney Company’s Disney Entertainment and ESPN Product & Technology team is seeking a senior data engineer to build and maintain Identity and Device data products. You will work with Databricks, Spark, and a modern data stack to deliver scalable data pipelines that power enterprise-wide customer identifiers and insights.

You will collaborate across squads, participate in on-call support, and contribute to architecture discussions while advancing data modeling and warehousing best

Qualifications

• 5+ years of related big data engineering experience modeling and developing large data pipelines.
• Hands-on experience with distributed systems such as Databricks compute using Spark, Presto, Hive to query and process large datasets at the petabyte level.
• Strong Python, Scala/Spark and SQL skills processing big datasets.
• Experience with at least one major MPP or cloud database technology (Snowflake, Redshift, BigQuery, Databricks).
• Familiarity with Data Modeling techniques and Data Warehousing standard methodologies and practices.
• Operational experience in multi-system production environment as point-of-contact / SME.
• Strong analytical and communication skills with problem-solving abilities.
• Solid experience with data integration and orchestration toolsets (e.g. Airflow), CI/CD.
• Solid experience with AWS S3.

Responsibilities

• Build and maintain product-driven initiatives to improve Identity and Device data product offerings.
• Enhance/improve the underlying codebase to be extensible, reusable, and maintainable; increase unit test coverage.
• Enhance/redesign data models with Architecture input to modernize products and improve performance and efficiency.
• Promote and support Agile methodologies such as Scrum, Kanban, and Scrumban; participate in ceremonies.
• Collaborate with squad, Product Managers, Designers, QA, Operations, and other stakeholders to share requirements and decisions.
• Participate in on-call support.
• Participate in code reviews and problem-solving.

Skills

Big data engineering

Python

Scala/Spark

SQL

Distributed systems

Problem solving

Communication

Data integration

Airflow

AWS S3

Education

Bachelor's degree in Computer Science / Information Systems / Software / Electrical or Electronics Engineering

Tools

Databricks

Spark

Presto

Hive

Snowflake

Redshift

BigQuery

Airflow

CI/CD

AWS S3

Job description

The Walt Disney Company’s Disney Entertainment and ESPN Product & Technology team is seeking a senior data engineer to build and maintain Identity and Device data products. You will work with Databricks, Spark, and a modern data stack to deliver scalable data pipelines that power enterprise-wide customer identifiers and insights.

You will collaborate across squads, participate in on-call support, and contribute to architecture discussions while advancing data modeling and warehousing best