Senior Data Engineer: Azure Databricks Lakehouse (Hybrid)

Harnham; Chicago (IL); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Harnham in Chicago is seeking an experienced Data Engineer to help lead a major Data & AI modernization initiative. You will shape a cloud-based data platform, build production-grade data pipelines, and establish engineering standards in a growing team.

The role emphasizes Azure Databricks, Spark, PySpark, Python, and SQL with a mix of on-premise migration and modern Lakehouse architectures. This hybrid role requires in-office days in Chicago.

Qualifications

• 6+ years of hands-on Data Engineering experience.
• Strong Azure Databricks and Spark/PySpark expertise in production environments.
• Advanced Python and SQL development skills.
• Experience owning end-to-end data pipelines from source through consumption.
• Strong ETL/ELT, data modeling, and Lakehouse architecture experience.
• Experience with CI/CD, testing, monitoring, orchestration, and production support.

Responsibilities

• Design and build end-to-end data pipelines from ingestion through transformation and downstream consumption.
• Develop scalable ETL/ELT solutions supporting both batch and streaming/CDC workloads.
• Build solutions using Python, SQL, PySpark, Spark, and Azure Databricks.
• Implement Lakehouse architectures and Bronze/Silver/Gold data models.
• Migrate and modernize legacy warehouse environments into a centralized cloud platform.
• Optimize data models, schemas, performance, and scalability.
• Establish CI/CD, automated testing, monitoring, alerting, and operational best practices.
• Troubleshoot complex production data issues and support platform reliability.
• Participate in code reviews and mentor junior-level engineers.

Skills

Data engineering

Data pipelines

ETL/ELT

Data modeling

CI/CD

Tools

Azure Databricks

Spark

PySpark

Python

SQL

Job description

Harnham in Chicago is seeking an experienced Data Engineer to help lead a major Data & AI modernization initiative. You will shape a cloud-based data platform, build production-grade data pipelines, and establish engineering standards in a growing team.

The role emphasizes Azure Databricks, Spark, PySpark, Python, and SQL with a mix of on-premise migration and modern Lakehouse architectures. This hybrid role requires in-office days in Chicago.