Senior Director, Software Engineering

BuddiesHR; United States; On-site; USD 145,000 - 190,000; Full time

Benefits offered by this job

Paid time off

Medical/dental/vision insurance

401(k)

Job summary

Circana is seeking a seasoned Data Engineer to design, build, and maintain robust data pipelines on the Azure cloud. You will leverage PySpark, Spark, and Airflow to process large-scale workloads with a focus on data quality, efficiency, and scalability.

You'll lead data-engineering efforts, collaborate across engineering and product teams, and implement CI/CD for data workflows while optimizing cloud costs.

Qualifications

• 8+ years in data engineering with Python, PySpark, Spark, and Airflow.
• Strong Python and SQL coding skills for scalable data apps.
• Experience with Git, Docker, Kubernetes, Terraform; DevOps practices.
• Experience leading teams and working in Agile/Scrum environments.

Responsibilities

• Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow.
• Implement real-time and batch data processing with Spark.
• Enforce data quality, governance, and security across the data lifecycle.
• Ensure data availability, reliability, and performance via monitoring.
• Manage cloud infrastructure and cost optimization for data workloads.
• Implement CI/CD pipelines for data workflows.
• Build and optimize large-scale data processing pipelines using Spark and PySpark.
• Implement data partitioning, caching, and performance tuning for Spark-based workloads.
• Work with diverse data formats to support advanced analytics and ML initiatives.
• Lead a team of data engineers and provide mentorship.
• Collaborate with peers across engineering, product, architects to drive scalable product.
• Partner with Program Management to align delivery timelines and mitigate risks.

Skills

PySpark

Spark

Airflow

Python

SQL

Git

Docker

Kubernetes

Terraform

AGILE/Scrum

Cloud (Azure/AWS/GCP)

AI tooling

ML Engineering

Team leadership

Communication

Job description

At Circana, we are fueled by our passion for continuous learning and growth, we seek and share feedback freely, and we celebrate victories both big and small in an environment that is flexible and accommodating to our work and personal lives. We have a global commitment to diversity, equity, and inclusion as we believe in the undeniable strength that diversity brings to our business, employees, clients, and communities. With us, you can always bring your full self to work. Join our inclusive, committed team to be a challenger, own outcomes, and stay curious together. Circana is proud to be Certified™ by Great Place To Work®. This prestigious award is based entirely on what current employees say about their experience working at Circana.

We are seeking a skilled and motivated Data Engineer to join our growing team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure on the Azure cloud platform. You will leverage your expertise in PySpark, Apache Spark, and Apache Airflow to process and orchestrate large-scale data workloads, ensuring data quality, efficiency, and scalability. If you have a passion for data engineering and a desire to make a significant impact, we encourage you to apply!

Job Responsibilities

• Design, develop, and optimize scalable DATA workflows using Python, PySpark, and Airflow
• Implement real-time and batch data processing using Spark
• Enforce best practices for data quality, governance, and security throughout the data lifecycle
• Ensure data availability, reliability and performance through monitoring and automation.
• Manage cloud infrastructure and cost optimization for data processing workloads
• Implement CI/CD pipelines for data workflows to ensure smooth and reliable deployments.
• Build and optimize large-scale data processing pipelines using Apache Spark and PySpark
• Implement data partitioning, caching, and performance tuning for Spark-based workloads.
• Work with diverse data formats (structured and unstructured) to support advanced analytics and machine learning initiatives.

Workflow Orchestration (Airflow)

• Design and maintain DAGs (Directed Acyclic Graphs) in Airflow to automate complex data workflows
• Monitor, troubleshoot, and optimize job execution and dependencies
• Lead a team of data engineers, providing technical guidance and mentorship
• Foster a collaborative environment and promote best practices for coding standards, version control, and documentation.
• Engage directly with clients to understand business needs, communicate technical concepts clearly.
• Collaborate with peers across engineering, product, architects to drive scalable and maintainable product
• Partner with the Program Management team to align delivery timelines, mitigate risks

Requirements

• 8+ years of experience in data engineering with expertise in Python, PySpark, Spark, and Airflow.
• Strong programming skills in Python, SQL with the ability to write efficient and maintainable code
• Deep understanding of Spark internals (RDDs, DataFrames, DAG execution, partitioning, etc.)
• Experience with Airflow DAGs, scheduling, and dependency management
• Knowledge of Git, Docker, Kubernetes, Terraform, and apply best practices of DevOps for CI/CD workflows
• Excellent problem-solving skills and ability to optimize large-scale data processing.
• Experience in leading teams and working in Agile/Scrum environments
• Experience with common data warehouse and/or data lake platform
• Knowledge of cloud technologies (Azure/AWS/Google)
• Proven experience using generative AI tools to accelerate software development, data engineering, system design, testing, and troubleshooting.
• Strong understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows.
• Ability to identify opportunities to apply AI solutions that improve team productivity, code quality, and operational efficiency.
• Experience of ML Engineering is a plus
• Strong communication and collaboration skills, with the ability to effectively work with remote teams

As well as the technical skills, experience and attributes that are required for the role, our shared behaviors sit at the core of our organization. Therefore, we always look for people who can continuously champion these behaviors throughout the business within their day-to-day role:

• Stay Curious: Being hungry to learn and grow, always asking the big questions.
• Seek Clarity: Embracing complexity to create clarity and inspire action.
• Own the Outcome: Being accountable for decisions and taking ownership of our choices.
• Center on the Client: Relentlessly adding value for our customers.
• Be a Challenger: Never complacent, always striving for continuous improvement.
• Champion Inclusivity: Fostering trust in relationships engaging with empathy, respect, and integrity.
• Commit to each other: Contributing to making Circana a great place to work for everyone.

Location

This position can be located in the following area(s): Remote or Hybrid to Chicago

Prospective candidates may be asked to consent to background checks (in accordance with local legislation and our candidate privacy notice) Your current employer will not be contacted without your permission.

The below range reflects the range of possible compensation for this role at the time of this posting. We may ultimately pay more or less than the posted range. This range may be modified in the future. An employee’s position within the salary range will be based on several factors including, but not limited to, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, shift, travel requirements, sales or revenue-based metrics, any collective bargaining agreements, and business or organizational needs. The salary range for this role is $145,000 to $190,000.

This job is also eligible for bonus pay.

We offer a comprehensive package of benefits including [paid time off, medical/dental/vision insurance and 401(k)] to eligible employees.

This role is subject to AI-assisted screening. Circana uses artificial intelligence (AI) to assess resumes for alignment with job requirements by helping locate details in resumes that relate to the job description.

This position is expected to remain open for approximately 30 days and may close earlier if sufficient qualified candidates are identified. #LI-SD1

Equal Opportunity Employer/Protected Veterans/Individuals with Disabilities
This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor.