Senior Azure Data Engineer - Databricks Lakehouse

Yoh, A Day & Zimmermann Company; Plano (TX); Hybrid; USD 77,000 - 110,000; Full time

Benefits offered by this job

Medical, Prescription, Dental & Vision

Health Savings Account (HSA)

401K Retirement Savings Plan

Direct Deposit & weekly epayroll

Certification and training

Job summary

Yoh, a Day & Zimmermann company, is partnering with a confidential enterprise to hire a Senior Azure Data Engineer with deep expertise in Azure Databricks to design, build, and optimize a modern lakehouse data platform in a hybrid Plano, TX setting.

You will develop scalable ETL/ELT pipelines, implement Delta Lake patterns, and enable advanced analytics in a high-volume environment, collaborating with data architects, data scientists, and cloud teams for secure, high-performance data solutions.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field or equivalent experience.
• 2+ years of data engineering experience, including 3+ years with Azure Databricks / Apache Spark (PySpark).
• Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns.
• Hands-on experience with ADLS Gen2, Event Hub, ADF, Key Vault, Synapse.
• Strong coding skills in Python and SQL; familiarity with Scala is a plus.
• Solid understanding of data modeling (OLTP/OLAP), star/snowflake schemas, and performance tuning.
• Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps.
• Databricks certification (e.g., Databricks Certified Data Engineer Professional).
• Streaming/real-time pipelines (Structured Streaming, Event Hub, Kafka).
• ML platform exposure (e.g., MLflow or supporting ML pipelines).
• Reporting/consumption layer experience (Power BI, Purview, Microsoft Fabric).
• Knowledge of governance/compliance standards (GDPR, HIPAA, SOC2).

Responsibilities

• Design and build scalable ETL/ELT pipelines using Azure Databricks (PySpark, Spark SQL).
• Develop ingestion workflows from multiple source systems into Bronze/Silver/Gold layers.
• Implement Delta Lake and Unity Catalog governance and schema evolution.
• Optimize pipeline performance and cost via partitioning, caching, clustering, and tuning.
• Integrate Databricks with ADLS Gen2, ADF, Synapse, Event Hub, Key Vault, and Azure DevOps.
• Implement secure access controls using managed identities, RBAC, and data masking.
• Partner with analytics and business teams to deliver trusted datasets, metrics, and KPIs.
• Provide technical leadership through mentoring, code reviews, and architectural guidance.
• Work with enterprise architecture and security teams to align with data privacy/security policies.
• Implement data quality checks, alerting, and lineage tracking with tools like Great Expectations and data observability platforms.
• Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines.

Skills

Python

SQL

Data engineering

Spark SQL

Education

Bachelor’s degree in Computer Science, Engineering, or related field

Tools

Azure Databricks

Delta Lake

Unity Catalog

Databricks notebooks

ADLS Gen2

Event Hub

ADF

Azure Synapse

Key Vault

Azure DevOps

Terraform

Databricks CLI

Great Expectations

Purview

Job description

Yoh, a Day & Zimmermann company, is partnering with a confidential enterprise to hire a Senior Azure Data Engineer with deep expertise in Azure Databricks to design, build, and optimize a modern lakehouse data platform in a hybrid Plano, TX setting.

You will develop scalable ETL/ELT pipelines, implement Delta Lake patterns, and enable advanced analytics in a high-volume environment, collaborating with data architects, data scientists, and cloud teams for secure, high-performance data solutions.