Senior Data Engineer: Azure Databricks Lakehouse Builder

Burtch Works; Chicago (IL); Hybrid; USD 148,000 - 164,000; Full time

Job summary

Burtch Works seeks a hands-on Senior Data Engineer in Chicago to design, build and maintain data pipelines and infrastructure powering analytics and ML across the firm. You will own pipelines end-to-end from ingestion to output in Databricks, writing code daily in Python, SQL and Spark.

This role requires 6+ years of data engineering experience, strong Databricks and cloud skills, and collaboration with stakeholders while upholding CI/CD and data governance practices.

Qualifications

• Bachelor's degree in CS/Engineering/Data Science or related field.
• 6+ years of hands-on data engineering experience.
• Proficiency in Python, SQL and Spark; live code interview.
• Experience with Azure Databricks and cloud data platforms.

Responsibilities

• Build end-to-end data solutions on Azure Databricks.
• Design, develop and maintain scalable ETL/ELT and streaming pipelines.
• Develop PySpark and Spark SQL transformations; optimize performance.
• Refactor and migrate data warehouses to the Databricks platform.
• Own pipeline orchestration, monitoring and CI/CD with testing and security checks.
• Implement data quality frameworks and monitoring.
• Troubleshoot Databricks platform and pipeline issues.
• Collaborate with cross-functional teams to align data work with business goals.
• Mentor junior engineers and promote best practices.

Skills

Python

SQL

Apache Spark

Databricks

Azure

ETL/ELT

Data Modeling

CI/CD

Git

PySpark

Education

Bachelor's degree in Computer Science, Engineering, Data Science or related field

Master's degree in Computer Science or Engineering

Tools

Azure Databricks

Delta Lake

ADLS Gen2

Terraform

Unity Catalog

MLflow

Notebooks

Job description

Burtch Works seeks a hands-on Senior Data Engineer in Chicago to design, build and maintain data pipelines and infrastructure powering analytics and ML across the firm. You will own pipelines end-to-end from ingestion to output in Databricks, writing code daily in Python, SQL and Spark.

This role requires 6+ years of data engineering experience, strong Databricks and cloud skills, and collaboration with stakeholders while upholding CI/CD and data governance practices.