Senior Data Engineering Director – Azure, Spark & Airflow

Circana; United States; Hybrid; USD 145,000 - 190,000; Full time

Benefits offered by this job

Bonus eligibility

Remote or Hybrid

Health benefits

Job summary

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on Azure using PySpark, Spark, and Airflow. You will ensure data quality, governance, and scalable performance across large workloads.

Responsibilities include leading a team of data engineers, collaborating with product and analytics, and implementing CI/CD pipelines for data workflows. Remote or Hybrid to Chicago with competitive compensation and bonus eligibility.

Qualifications

• 8+ years of data engineering with Python, PySpark, Spark and Airflow.
• Strong Python & SQL coding for scalable data pipelines.
• Experience leading teams in Agile environments.

Responsibilities

• Design, develop, and optimize scalable DATA workflows using Python, PySpark, and Airflow.
• Implement real-time and batch data processing using Spark.
• Enforce best practices for data quality, governance, and security throughout the data lifecycle.
• Lead data pipelines reliability, monitoring, and automation.

Skills

Python

PySpark

Spark

Airflow

SQL

Git

Docker

Kubernetes

Agile/Scrum

Leadership

Communication

Tools

Azure

Terraform

Job description

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on Azure using PySpark, Spark, and Airflow. You will ensure data quality, governance, and scalable performance across large workloads.

Responsibilities include leading a team of data engineers, collaborating with product and analytics, and implementing CI/CD pipelines for data workflows. Remote or Hybrid to Chicago with competitive compensation and bonus eligibility.