Senior Data Engineering Director — Azure, Spark & Airflow

BuddiesHR; United States; Hybrid; USD 145,000 - 190,000; Full time

Benefits offered by this job

Paid time off

Medical/dental/vision insurance

401(k)

Job summary

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on the Azure cloud. You will leverage PySpark, Spark, and Airflow to process large-scale workloads with a focus on data quality, efficiency, and scalability.

You'll lead data-engineering efforts, collaborate across engineering and product teams, and implement CI/CD for data workflows while optimizing cloud costs.

Qualifications

• 8+ years in data engineering with Python, PySpark, Spark, and Airflow.
• Strong Python and SQL coding skills for scalable data apps.
• Experience with Git, Docker, Kubernetes, Terraform; DevOps practices.
• Experience leading teams and working in Agile/Scrum environments.

Responsibilities

• Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
• Implement real-time and batch data processing with Spark.
• Enforce data quality, governance, and security across the data lifecycle.
• Ensure data availability, reliability, and performance via monitoring.
• Manage cloud infrastructure and cost optimization for data workloads.
• Implement CI/CD pipelines for data workflows.
• Build and optimize large-scale data processing pipelines using Spark and PySpark.
• Implement data partitioning, caching, and performance tuning for Spark-based workloads.
• Work with diverse data formats to support advanced analytics and ML initiatives.
• Lead a team of data engineers and provide mentorship.
• Collaborate with peers across engineering, product, architects to drive scalable product.
• Partner with Program Management to align delivery timelines and mitigate risks.

Skills

PySpark

Spark

Airflow

Python

SQL

Git

Docker

Kubernetes

Terraform

AGILE/Scrum

Cloud (Azure/AWS/GCP)

AI tooling

ML Engineering

Team leadership

Communication

Job description

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on the Azure cloud. You will leverage PySpark, Spark, and Airflow to process large-scale workloads with a focus on data quality, efficiency, and scalability.

You'll lead data-engineering efforts, collaborate across engineering and product teams, and implement CI/CD for data workflows while optimizing cloud costs.