Senior Data Engineer (Remote)

Circana Careers; Cape Town; Remote; ZAR 600,000 - 900,000; Full time

Job summary

A leading global data solutions provider is seeking a skilled Data Engineer to design and maintain robust data pipelines. Ideal candidates will have hands-on experience with Azure, PySpark, and Apache Spark, along with strong collaboration skills. Join a dynamic team dedicated to continuous learning and growth, and play a pivotal role in data engineering projects that impact clients globally.

Qualifications

• Proven experience in data engineering with hands-on expertise in Azure Data Services, PySpark, Apache Spark and Apache Airflow.
• Strong programming skills in Python and SQL.
• Deep understanding of Spark internals including RDDs, DataFrames, and performance optimization.

Responsibilities

• Design and develop efficient and scalable ETL / ELT pipelines using Python, PySpark and Apache Airflow.
• Manage and optimize cloud infrastructure (Azure) for data processing workloads.
• Implement and maintain CI / CD pipelines for data workflows.

Skills

Communication

Collaboration

Python

SQL

PySpark

Apache Spark

Apache Airflow

Problem-solving

Tools

Azure Data Services

Docker

Kubernetes

CI/CD

Job description

Overview

At Circana we are fueled by a passion for continuous learning and growth. We seek and share feedback freely and celebrate victories both big and small in an environment that is flexible and accommodating to work and personal lives. We are a global company dedicated to fostering inclusivity and belonging. We value the unique experiences, cultures, and viewpoints that each individual brings. By embracing a wide range of backgrounds, skills, expertise, and beyond, we create a stronger, more innovative environment for our employees, clients and communities. Circana is proud to be Certified by Great Place To Work, based on what current employees say about their experience working at Circana.

Learn more at .

What will you be doing

We are seeking a skilled and motivated Data Engineer to join a growing global team. In this role you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure on the Azure cloud platform. You will leverage your expertise in PySpark, Apache Spark, and Apache Airflow to process and orchestrate large-scale data workloads, ensuring data quality, efficiency and scalability. If you have a passion for data engineering and a desire to make a significant impact, we encourage you to apply!

Job Responsibilities

• ETL / ELT Pipeline Development: Design, develop and optimize efficient and scalable ETL / ELT pipelines using Python, PySpark and Apache Airflow.
• Implement batch and real-time data processing solutions using Apache Spark.
• Ensure data quality, governance and security throughout the data lifecycle.
• Cloud Data Engineering: Manage and optimize cloud infrastructure (Azure) for data processing workloads with a focus on cost-effectiveness.
• Implement and maintain CI / CD pipelines for data workflows to ensure smooth and reliable deployments.
• Big Data & Analytics: Develop and optimize large-scale data processing pipelines using Apache Spark and PySpark.
• Implement data partitioning, caching and performance tuning techniques to enhance Spark-based workloads.
• Work with diverse data formats (structured and unstructured) to support advanced analytics and machine learning initiatives.
• Workflow Orchestration (Airflow): Design and maintain DAGs (Directed Acyclic Graphs) in Apache Airflow to automate complex data workflows.
• Monitor, troubleshoot and optimize job execution and dependencies within Airflow.
• Team Leadership & Collaboration: Provide technical guidance and mentorship to a team of data engineers in India.
• Foster a collaborative environment and promote best practices for coding standards, version control and documentation.

Qualifications

• Client-facing role, so strong communication and collaboration skills are vital.
• Proven experience in data engineering with hands-on expertise in Azure Data Services, PySpark, Apache Spark and Apache Airflow.
• Strong programming skills in Python and SQL with the ability to write efficient and maintainable code.
• Deep understanding of Spark internals including RDDs, DataFrames, DAG execution, partitioning and performance optimization techniques.
• Experience with designing and managing Airflow DAGs, scheduling and dependency management.
• Knowledge of CI / CD pipelines, containerization technologies (Docker, Kubernetes) and DevOps principles applied to data workflows.
• Excellent problem-solving skills and a proven ability to optimize large-scale data processing tasks.
• Prior experience in leading teams and working in Agile / Scrum development environments.
• A track record of working effectively with global remote teams.

Desirable

• Experience with data modelling and data warehousing concepts.
• Familiarity with data visualization tools and techniques.
• Knowledge of machine learning algorithms and frameworks.

Circana Behaviours

As well as the technical skills, experience and attributes required for the role, our shared behaviours sit at the core of our organization. We look for people who can champion these behaviours in day-to-day roles:

• Stay Curious: Always eager to learn and grow, asking the big questions.
• Seek Clarity: Embrace complexity to create clarity and inspire action.
• Own the Outcome: Be accountable for decisions and take ownership of our choices.
• Center on the Client: Relentlessly add value for our customers.
• Be a Challenger: Never complacent, always strive for continuous improvement.
• Champion Inclusivity: Foster trust, engage with empathy, respect and integrity.
• Commit to Each Other: Contribute to making Circana a great place to work for everyone.

Location

This position can be located in the following area(s): Bracknell

LI-KM1

Experience

Senior IC

Key Skills

Apache Hive, S3, Hadoop, Redshift, Spark, AWS, Apache Pig, NoSQL, Big Data, Data Warehouse, Kafka, Scala

Employment Type

Full-Time

Department / Functional Area

Data Engineering

Experience

years

Vacancy

1