Sr. Data Engineer

The Walt Disney Company; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

The Walt Disney Company Bengaluru is seeking an experienced Data Engineer to architect and grow data products and pipelines using Scala and Python. You will build scalable ETL processes in Big Data environments, implement Lakehouse architectures, and collaborate with Data Product Managers and Data Engineers to deliver robust data solutions.

You will maintain data governance documentation, ensure SLA adherence, and advocate agile practices while proactively solving problems and automating

Qualifications

• Bachelor's degree or equivalent work experience in CS/IS/Engineering.
• At least 5 years of data engineering experience building large data pipelines.
• Proficient with cloud data platforms and ETL pipelines.
• Strong SQL and data modeling skills.
• Hands-on with distributed systems (Spark/Hadoop) and databricks workflows.
• Experience with Airflow and data lakehouse concepts.
• Familiar with Scrum/Agile methodologies.
• Strong communication and cross-functional collaboration.

Responsibilities

• Contribute to architecture, design, and growth of data products and pipelines.
• Develop scalable ETL pipelines in Big Data environments (cloud/on-prem/hybrid).
• Implement Lakehouse architecture with key partners.
• Collaborate with Data Product Managers and Engineers to deliver data solutions.
• Maintain documentation for data governance and quality.
• Ensure uptime and SLA adherence for data platforms.
• Advocate agile/scrum practices and continuous improvement.
• Research and solve new data engineering challenges.
• Automate processes and optimize costs where possible.
• Engage with customers to prioritize platform improvements.

Skills

SQL

Spark

Python

Scala

Hadoop

Airflow

Databricks

Snowflake

Redshift

BigQuery

AWS

Agile/Scrum

Problem Solving

Education

Bachelor's degree in CS/IS/Eng

Tools

Airflow

Databricks

Snowflake

Redshift

BigQuery

Spark

Hadoop

Hive/Presto

PySpark

AWS

Job description

Responsibilities:

• Contribute to the architecture, design, and growth of our Data Products and Data pipelines in Scala and Python / Spark while maintaining uptime SLAs.
• Develop scalable solutions, building ETL pipelines in Big Data environments (cloud, on-prem, hybrid).
• Implement the Lakehouse architecture, working with key partners to shift towards a Lakehouse-centric data platform.
• Collaborate with Data Product Managers, Data Architects, and Data Engineers to design, implement, and deliver successful data solutions.
• Maintain detailed documentation of your work and changes to support data quality and data governance.
• Ensure high operational efficiency and quality of your solutions to meet SLAs and support commitment to our customers (Data Science, Data Analytics teams).
• Be an active participant and advocate of agile/scrum practice to ensure health and process improvements for your team.
• Be a problem solver; when presented with new challenges, you are expected to research and network to find solutions.
• Seek out answers to business problems and look for opportunities to automate processes and optimize costs.
• Engage with and understand our customers, forming relationships that allow us to understand and prioritize both innovative new offerings and
• incremental platform improvements.

Requirements:

• Bachelor's degree in computer science, Information Systems, Software, Electrical or Electronics Engineering, or a comparable field of study, and/or equivalent work experience.
• At least 5 years of data engineering experience developing large data pipelines.
• Strong hands-on Experience with Cloud technologies like AWS (S3 EMR, EC2) and building data pipelines.
• Strong algorithmic problem-solving expertise.
• Strong SQL skills and ability to create queries to extract data and build performant datasets.
• Hands-on experience with distributed systems such as Spark, Hadoop (HDFS, Hive, Presto, PySpark) to query and process data.
• Strong fundamental Scala and Python programming skills.
• Experience with at least one major MPP or cloud database technology (Snowflake, Redshift, BigQuery).
• Solid experience with data integration toolsets (i. e, Airflow) and writing and maintaining Data Pipelines with Databricks.
• Experience in Data Modeling techniques and Data Warehousing standard methodologies and practices.
• Familiar with Scrum and Agile methodologies.
• Demonstrated excellent interpersonal and communication skills, including the ability to partner with others and build consensus in a cross-functional team toward a desired outcome.

Preferred Qualifications:

• Experience with at least one major Massively Parallel Processing (MPP) or cloud database technology (Snowflake, Redshift, Big Query).
• Experience with Software Application development.
• Good data acumen to understand domain /functional knowledge and to be able to articulate analysis of data points in a business context for stakeholders.
• Experience working with a test-first mentality.
• Deep Understanding of AWS or other cloud providers, as well as infrastructure as code.

Tech stack:

• AWS
• Snowflake
• Spark
• Databricks
• Delta Lake
• Airflow
• Python
• Scala