Senior Director, Data Engineering & Cloud Analytics

Circana; Chicago (IL); Hybrid; USD 145,000 - 190,000; Full time

Benefits offered by this job

Bonus pay

Benefits package

Flexible work

Job summary

Circana seeks a Senior Director of Software Engineering to lead data engineering on the Azure cloud, designing and scaling data pipelines with PySpark, Spark, and Airflow. You will guide engineers and partner with product and analytics teams to deliver scalable solutions.

You will drive data quality, governance, and performance while implementing CI/CD for data workloads and mentoring the team in Agile practices. Hybrid/remote options available.

Qualifications

• 8+ years of data engineering experience with Python, PySpark, Spark, and Airflow.
• Strong Python and SQL coding skills.
• Deep understanding of Spark internals (RDDs, DataFrames, DAG execution, partitioning).
• Experience with Airflow DAGs, scheduling, and dependency management.
• Knowledge of Git, Docker, Kubernetes, Terraform; DevOps CI/CD practices.
• Experience leading teams in Agile/Scrum environments.

Responsibilities

• Data Engineering & Data Pipeline Development: design, develop, and optimize scalable DATA workflows using Python, PySpark, and Airflow.
• Cloud Data Engineering: manage cloud infrastructure and cost optimization for data processing workloads.
• Big Data & Analytics: build and optimize large-scale data processing pipelines using Spark and PySpark.
• Workflow Orchestration (Airflow): design and maintain DAGs to automate complex data workflows.
• Team Leadership & Collaboration: lead a team of data engineers, mentor, collaborate across engineering, product, and analytics.

Skills

Python

PySpark

Spark

Airflow

SQL

Leadership

Cloud computing

Tools

Git

Docker

Kubernetes

Terraform

Airflow

Spark

Azure

AWS

Google Cloud

CI/CD

Job description

Circana seeks a Senior Director of Software Engineering to lead data engineering on the Azure cloud, designing and scaling data pipelines with PySpark, Spark, and Airflow. You will guide engineers and partner with product and analytics teams to deliver scalable solutions.

You will drive data quality, governance, and performance while implementing CI/CD for data workloads and mentoring the team in Agile practices. Hybrid/remote options available.