Senior Data Engineer: Azure Databricks & ETL Lead

McKinsey & Company; Dallas (TX); On-site; USD 195,300 - 199,200; Full time

Benefits offered by this job

Competitive salary

Comprehensive benefits package

Bonuses possible

Mentorship & growth programs

Job summary

McKinsey & Company's Periscope by McKinsey team is seeking a senior data engineer to design, build, and scale data pipelines on Azure using Databricks. You will lead projects end‑to‑end, mentor engineers, and work with client‑facing teams to implement enterprise solutions that meet business goals.

Expect a role that combines hands‑on development with strategic insight, optimizing performance and cost in large data environments, and applying best practices for data governance, security, and

Qualifications

• Bachelor's or master's degree in computer science, engineering or a related field.
• 6+ years of experience in data engineering with a strong background in big data technologies.
• Proficiency in Databricks on Azure, Spark, Python, and Spark SQL.
• Extensive experience with Delta Live Tables, Unity Catalog, Delta Sharing, and SQL Warehouse (server and serverless).
• Experience in handling large data volumes and orchestrating automated ETL using Azure Data Factory, Airflow / data pipelines using CI/CD and Cloud Technologies.
• Development efficiency leveraging AI framework technologies.
• Demonstrated ability to optimize performance and cost in large‑scale data environments.
• Proven track record of leading projects independently and mentoring junior team members.
• Experience with cloud platforms, particularly Azure, and a solid understanding of cloud‑native architectures.
• Experience adopting software development best practices (e.g., modularization, testing, refactoring, etc.).
• Experience with data governance, security, and compliance in a cloud environment.
• Familiarity in understanding BI framework

Responsibilities

• Design, develop, and maintain scalable data pipelines and systems using Databricks on Azure.
• Implement and optimize ETL processes with a focus on performance, cost‑efficiency, and reliability.
• Utilize advanced Databricks technologies (DLT, Unity Catalog, Delta Sharing, SQL Warehouse) to enhance data management and sharing capabilities.
• You will be leading and executing data engineering projects from inception to completion, ensuring timely delivery and high quality while you continuously monitor, troubleshoot, and improve data pipelines and workflows to ensure optimal performance and cost-effectiveness.
• You will collaborate with cross‑functional teams to understand data needs and deliver solutions that meet business requirements, mentor, and educate junior engineers on best practices, emerging technologies, and efficient data engineering techniques.
• You will also work with client‑facing teams to help solve and deploy client specific solutions that integrate into the client’s internal systems.

Skills

Databricks on Azure

Spark

Python

Spark SQL

Delta Live Tables

Unity Catalog

Delta Sharing

SQL Warehouse

Azure Data Factory

Airflow

CI/CD

Cloud technologies

AI framework technologies

Data governance & security

BI framework understanding

Mentoring junior team members

Leading projects independently

Education

Bachelor's or master's degree in computer science, engineering or a related field

Tools

Databricks

Airflow

Delta Live Tables

Unity Catalog

Delta Sharing

SQL Warehouse

Job description

McKinsey & Company's Periscope by McKinsey team is seeking a senior data engineer to design, build, and scale data pipelines on Azure using Databricks. You will lead projects end‑to‑end, mentor engineers, and work with client‑facing teams to implement enterprise solutions that meet business goals.

Expect a role that combines hands‑on development with strategic insight, optimizing performance and cost in large data environments, and applying best practices for data governance, security, and