Remote Senior Director, Data Engineering

Circana; Chicago (IL); Hybrid; USD 120,000 - 165,000; Full time

Benefits offered by this job

Bonus eligibility

Benefits package (PTO, health, 401(k))

Job summary

Circana is hiring a Senior Director, Software Engineering to design, build, and lead scalable data pipelines on Azure, leveraging PySpark and Airflow. You will guide a team of data engineers, collaborate with product and architects, and own delivery outcomes in a remote/hybrid setup to Chicago.

The role emphasizes data quality, security, and cost optimization, with CI/CD practices and strong cloud knowledge across Azure/AWS/Google. Excellent communication and leadership are essential.

Qualifications

• 8+ years of experience in data engineering with Python, PySpark, Spark, and Airflow.
• Strong Python/SQL programming; write efficient, maintainable code.
• Deep understanding of Spark internals (RDDs, DataFrames, DAGs, partitions).
• Experience with Airflow DAGs, scheduling, dependencies.
• Knowledge of Git, Docker, Kubernetes, Terraform; CI/CD DevOps practices.
• Experience leading teams in Agile/Scrum environments.
• Experience with data warehouse or data lake platforms.
• Knowledge of cloud technologies (Azure/AWS/Google).
• Experience using generative AI tools to accelerate development.
• Strong communication and collaboration; ability to work with remote teams.
• AI/ML engineering experience is a plus.

Responsibilities

• Design, build, and maintain robust data pipelines and infrastructure on the Azure cloud platform.
• Lead a team of data engineers; provide technical guidance and mentorship.
• Collaborate with product, architects, and clients to align delivery timelines and expectations.
• Implement CI/CD pipelines for data workflows to ensure reliable deployments.
• Monitor data quality, governance, security, and performance across data lifecycles.
• Manage cloud infrastructure and cost optimization for data workloads.
• Develop real-time and batch data processing using Spark and PySpark.

Skills

Team leadership

Communication

Agile/Scrum

Problem solving

Strategic thinking

Tools

Python

PySpark

Spark

Airflow

Git

Docker

Kubernetes

Terraform

Azure

AWS

Google Cloud

Job description

Circana is hiring a Senior Director, Software Engineering to design, build, and lead scalable data pipelines on Azure, leveraging PySpark and Airflow. You will guide a team of data engineers, collaborate with product and architects, and own delivery outcomes in a remote/hybrid setup to Chicago.

The role emphasizes data quality, security, and cost optimization, with CI/CD practices and strong cloud knowledge across Azure/AWS/Google. Excellent communication and leadership are essential.