Senior Director of Data Engineering - Spark & Airflow

Circana; Illinois; Hybrid; USD 120,000 - 165,000; Full time

Benefits offered by this job

Bonus pay

Paid time off

Medical/dental/vision insurance

401(k)

Job summary

Circana is seeking a skilled Data Engineer to design, build, and maintain robust data pipelines on the Azure/AWS/GCP cloud. You will use PySpark, Spark, and Airflow to process large datasets, ensure data quality, and optimize performance.

You will lead a team of data engineers, collaborate with product teams, and interact with clients to translate business needs into scalable solutions. Remote or hybrid to Chicago with competitive benefits.

Qualifications

• 8+ years of data engineering experience with Python, PySpark, Spark, and Airflow.
• Strong Python and SQL programming ability.
• Understanding of Spark internals and DAG execution.
• Experience with Airflow DAGs, scheduling, and dependencies.
• Knowledge of Git, Docker, Kubernetes, Terraform, and DevOps for CI/CD.
• Experience with data warehouses/lands and cloud tech (Azure/AWS/Google).
• Familiarity with AI-assisted engineering and ML engineering is a plus.
• Excellent communication and collaboration, including remote work.

Responsibilities

• Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
• Implement real-time and batch data processing with Spark.
• Enforce data quality, governance, and security across the data lifecycle.
• Ensure data availability, reliability, and performance via monitoring and automation.
• Manage cloud infrastructure and cost optimization for processing workloads.
• Build and maintain Airflow DAGs to automate complex data workflows.
• Lead a team of data engineers and collaborate with product and architecture teams.
• Engage with clients to understand business needs and communicate technical concepts.

Skills

Python

PySpark

Spark

Airflow

SQL

Git

Docker

Kubernetes

Terraform

DevOps

Agile/Scrum

Data warehouse

Data lake

Azure

AWS

Google Cloud

Generative AI tools

ML Engineering

Tools

Airflow

Spark

PySpark

Git

Docker

Kubernetes

Terraform

CI/CD

Job description

Circana is seeking a skilled Data Engineer to design, build, and maintain robust data pipelines on the Azure/AWS/GCP cloud. You will use PySpark, Spark, and Airflow to process large datasets, ensure data quality, and optimize performance.

You will lead a team of data engineers, collaborate with product teams, and interact with clients to translate business needs into scalable solutions. Remote or hybrid to Chicago with competitive benefits.