Senior Azure Data Engineer

Motion Recruitment; Chicago (IL); On-site; USD 140,000 - 190,000; Full time

Job summary

A leading healthcare performance improvement firm is seeking a Sr. Azure Data Engineer in Chicago. The ideal candidate will have over 5 years of experience in data engineering and strong skills in designing and supporting data pipelines. Proficiency in Azure Data Factory, PySpark, and SQL is essential. This role requires hands-on experience with cloud environments and a commitment to delivering quality results. This position supports a hybrid work model with three days onsite.

Qualifications

• 5+ years' experience as a data engineer.
• 3+ years with ETL/ELT concepts and troubleshooting.
• 2+ years of production-quality Python development.
• 1+ years using Azure Data Factory (ADF) and Databricks.
• PySpark experience with distributed transformations, CICD, and testing.
• 1+ years using Azure Data Factory (ADF) to build/operate pipelines, integration with Databricks/ADLS.
• 1+ year using Databricks for notebooks/jobs/workflows, Delta Lake patterns, and basic cluster configuration.
• Azure Fundamentals + Pulumi: managing ADLS Gen2, Azure Portal, Resource Groups, Azure SQL, and Pulumi scripts.

Responsibilities

• Design and support end-to-end data pipelines.
• Optimize SQL and PySpark transformations.
• Create and maintain Azure Data Factory pipelines.
• Provision Azure components using Pulumi.
• Provision and maintain Azure components using Pulumi (Infrastructure as Code).
• Participate in code reviews, documentation, and operational support (triage + root cause analysis).

Skills

Data engineering

ETL/ELT concepts

SQL advanced querying

Python development

PySpark optimization

Azure Data Factory

Databricks workflows

Infrastructure as Code (Pulumi)

Tools

Azure SQL

ADLS Gen2

Python testing frameworks

Pulumi

Pulumi

Azure SQL

Delta Lake

Job description

Our client, the nation’s largest provider-driven healthcare performance improvement company, is looking for a Sr. Azure Data Engineer with solid, real-time hands-on experience in:

• PySpark
• Databricks (notebooks, jobs, workflows, External/Managed tables, SQL Warehouse, AI/ML)
• ADLS (Blob Storage)

Location: Downtown Chicago/60606 (3 days onsite Tues, Wed, Thurs)

What you will be doing:

Key Responsibilities

• Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing).
• Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
• Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling).
• Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations).
• Provision and maintain Azure components using Pulumi (Infrastructure as Code).
• Participate in code reviews, documentation, and operational support (triage + root cause analysis).

Must-have Skills (Required)

• 5+ years' experience as a data engineer
• 3+ years doing ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.
• 3+ years in SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation.
• 2+ years using Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage).
• PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage.
• 1+ years using Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS.
• 1+ year using Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration.
• Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments.

Nice-to-have Skills

• R: Ability to support/translate validation rules with SQL scripts and create data quality reports.
• TypeScript: Useful for pulumi pipeline to create Azure components.
• Java: Useful for integration with existing services/components.
• .NET: Useful for integration with existing services/components.
• Angular / Spring Boot: Minor troubleshooting or coordination with app teams.