Lead Data Engineer - Databricks Lakehouse

U.S. Bank; Atlanta (GA); Hybride; USD 133 000 - 157 000; Plein temps

Avantages offerts par ce poste

Healthcare

401(k) plan

Paid vacation

11 paid holidays

Adoption assistance

Sick leave

Résumé du poste

U.S. Bank is seeking a senior Data Engineer to lead the design and delivery of scalable, governed data products on a Databricks Lakehouse across Azure and AWS.

You will provide hands-on technical leadership, guide engineering standards, and collaborate with product, architecture, governance, and business stakeholders to deliver trusted data for analytics and AI use cases. The role requires strong Python, PySpark, SQL skills, and extensive experience with Databricks, Delta Lake, and cloud data

Qualifications

• Bachelor’s degree, or equivalent work experience.
• Six to eight years of relevant experience.
• Strong hands-on experience with Python, PySpark, SQL, Spark SQL, and ETL/ELT development.
• Hands-on expertise with Databricks, Delta Lake, Unity Catalog, medallion architecture, and Databricks Workflows/Lakeflow.
• Experience building batch, CDC, streaming, API, and file-based pipelines, including Kafka or Azure Event Hubs.
• Strong cloud data engineering experience with Azure and/or AWS, including ADLS Gen2, Data Factory, S3, IAM, and related technologies.
• Knowledge of Snowflake, Delta Sharing, Delta UniForm, Apache Iceberg, and cross-cloud data sharing.
• Experience with data modeling, Power BI/DAX, data quality, governance, lineage, and PII/PCI controls.
• Strong knowledge of Git, CI/CD, automated testing, technical design, architecture reviews, and engineering best practices.
• Demonstrated ability to provide technical direction through influence rather than direct authority.

Responsabilités

• Lead and guide Databricks data engineering across Bronze, Silver, and Gold layers, focusing on trusted, reusable Silver and Gold data products.
• Translate business requirements into scalable data architecture, engineering designs, delivery standards, and technical acceptance criteria.
• Remain hands-on in engineering and development, including coding, code review, and code management, while guiding batch, CDC, streaming, API, and file-based pipelines.
• Review data models, solution designs, and critical code for quality, security, scalability, performance, and alignment with engineering standards.
• Drive cross-cloud data integration and sharing across Azure, AWS, Databricks, and Snowflake.
• Embed data quality, lineage, governance, PII and PCI controls, monitoring, and reliability into data solutions.
• Establish engineering best practices for CI/CD, testing, source control, deployment, and observability.
• Provide technical leadership through influence across distributed onshore, offshore, and GCC engineering teams, without direct people-management responsibility.
• Partner with product, architecture, governance, platform, and business stakeholders to resolve technical dependencies, risks, and delivery challenges.

Connaissances

Python

PySpark

SQL

Spark SQL

ETL/ELT development

Formation

Bachelor’s degree, or equivalent work experience

Outils

Databricks

Delta Lake

Unity Catalog

medallion architecture

Databricks Workflows/Lakeflow

Kafka

Azure Event Hubs

Azure ADLS Gen2

Data Factory

S3

IAM

Description du poste

U.S. Bank is seeking a senior Data Engineer to lead the design and delivery of scalable, governed data products on a Databricks Lakehouse across Azure and AWS.

You will provide hands-on technical leadership, guide engineering standards, and collaborate with product, architecture, governance, and business stakeholders to deliver trusted data for analytics and AI use cases. The role requires strong Python, PySpark, SQL skills, and extensive experience with Databricks, Delta Lake, and cloud data