Senior Director, Data Engineering & AI Platforms

BuddiesHR; Chicago (IL); Hybrid; USD 120,000 - 165,000; Full time

Benefits offered by this job

Bonus pay

401(k) plan

Medical/dental/vision insurance

Job summary

Circana is seeking a skilled Data Engineer to design, build, and maintain robust data pipelines on Azure. You will leverage PySpark, Spark, and Airflow to process large-scale workloads, ensure data quality, and optimize performance.

You will lead data workflows, collaborate with product and engineering teams, and apply best practices for governance, security, and scalable data architectures. Remote or Hybrid to Chicago options are available.

Qualifications

• 8+ years of experience in data engineering with Python, PySpark, Spark, and Airflow.
• Strong programming skills in Python and SQL with maintainable code.
• Knowledge of cloud technologies (Azure/AWS/Google).
• Experience with Git, Docker, Kubernetes, Terraform and CI/CD workflows.

Responsibilities

• Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
• Implement real-time and batch data processing using Spark.
• Enforce data quality, governance, and security across the data lifecycle.
• Ensure data availability, reliability and performance via monitoring and automation.
• Manage cloud infrastructure and cost optimization for data workloads.
• Implement CI/CD pipelines for data workflows for reliable deployments.
• Build and optimize large-scale data processing pipelines with Spark and PySpark.
• Work with structured and unstructured data to support analytics and ML initiatives.

Skills

Python

PySpark

Spark

Airflow

Git

Docker

Kubernetes

Terraform

Tools

Azure

AWS

Google Cloud

Job description

Circana is seeking a skilled Data Engineer to design, build, and maintain robust data pipelines on Azure. You will leverage PySpark, Spark, and Airflow to process large-scale workloads, ensure data quality, and optimize performance.

You will lead data workflows, collaborate with product and engineering teams, and apply best practices for governance, security, and scalable data architectures. Remote or Hybrid to Chicago options are available.