Senior Director of Data Platform & AI - Remote

BuddiesHR; Chicago (IL); Hybrid; USD 120,000 - 165,000; Full time

Benefits offered by this job

Bonus pay

401(k)

Medical/dental/vision insurance

Paid time off

Job summary

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on Azure, leveraging PySpark, Spark, and Airflow. You will lead data projects, collaborate with engineering and product teams, and drive scalable data solutions across the organization.

The role requires 8+ years in data engineering, strong Python/SQL, and hands-on experience with Spark, Airflow, and cloud platforms.

Qualifications

• 8+ years of data engineering experience with Python, PySpark, Spark, and Airflow.
• Strong Python and SQL coding skills.
• Deep Spark internals and DAG execution knowledge.
• Experience with Airflow DAGs and CI/CD for data workflows.
• DevOps practices: Git, Docker, Kubernetes, Terraform.
• Leadership experience in Agile/Scrum teams.
• Experience with data warehouses or data lakes.
• Cloud experience: Azure/AWS/Google.
• Experience with AI-assisted engineering and ML is a plus.
• Strong communication and remote collaboration.

Responsibilities

• Design, develop, and optimize scalable DATA workflows using Python, PySpark, and Airflow.
• Implement real-time and batch data processing using Spark.
• Enforce data quality, governance, and security across the data lifecycle.
• Ensure data availability, reliability and performance through monitoring and automation.

Skills

Python

PySpark

Spark

Airflow

SQL

Git

Docker

Kubernetes

Terraform

DevOps practices

Agile/Scrum

Communication

Remote collaboration

AI tools

Tools

Azure

AWS

Google Cloud

Job description

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on Azure, leveraging PySpark, Spark, and Airflow. You will lead data projects, collaborate with engineering and product teams, and drive scalable data solutions across the organization.

The role requires 8+ years in data engineering, strong Python/SQL, and hands-on experience with Spark, Airflow, and cloud platforms.