Senior Data Engineer

X, bigly labs; Johannesburg; On-site; ZAR 1,200,000 - 1,800,000; Full time

Job summary

X, bigly labs in Johannesburg seeks a Senior Data Engineer to lead the development of a modern, scalable, and secure data platform that powers analytics, machine learning, and AI initiatives. You will design, build, and optimise robust data pipelines to enable governed data sharing with external partners.

You will collaborate with data analysts, data scientists, and AI engineers to ensure high-performance platforms supporting healthcare, retail, and insurance innovations.

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Certification in cloud platform: AWS, Azure Data Engineer, or Databricks Certified Data Engineer.
• 6+ years of experience in data engineering or software development.
• 3+ years with Databricks Intelligence Platform and Medallion Architecture.
• Strong proficiency in Python, SQL, Spark, and cloud-native data services.
• Familiarity with cloud-native data integration tools (e.g. SAP Datasphere, Snowflake Airflow).
• Experience with GitHub, JIRA, Confluence and MIRO.
• Proven experience in data platforms for data science, analytics, and AI workloads.
• ETL development with CI/CD, data governance, and data quality frameworks.
• Solid understanding of data science workflows and AI concepts.

Responsibilities

• Lead development of a scalable and secure data platform using Medallion Architecture for analytics, data science, and AI workloads.
• Build robust, modular data pipelines with Python, SQL, Spark, and cloud-native services.
• Integrate diverse data sources with scalable ingestion frameworks and APIs.
• Collaborate with analysts, data scientists, AI engineers, and data quality analysts.
• Ensure data integrity, privacy, and compliance with automated data quality checks and lineage.
• Document architecture, data flows, and standards in version-controlled repos.

Skills

Python

SQL

Spark

Cloud platforms

Databricks

Medallion Architecture

CI/CD for ETL

Data governance

Data integration tools

GitHub/JIRA/Confluence/MIRO

Education

Bachelor’s degree in Computer Science or related

Tools

Databricks

Apache Spark

AWS

Azure

GCP BigQuery

Snowflake

Airflow

CI/CD tooling

Job description

We are X, bigly labs, Dis-Chem’s high-performance innovation hub; a place where bold ideas meet data, design, and radical customer focus to reimagine what healthcare can be.

Our mission is to power the future of healthcare by lowering costs, unlocking new possibilities, and improving access to healthcare for all South Africans. We do this through cutting-edge digital solutions that deliver smarter, more human, and truly patient-focused experiences. Here, your work is not limited to a whiteboard; it becomes real. It drives decisions, shapes experiences, and moves healthcare forward. We are driven by one big question: How do we use data and today’s technology to build breakthroughs to better health?

There’s only one question left: are you ready to make healthcare smarter, and actually mean it?

We are seeking a Senior Data Engineer to lead the development of a modern, scalable, and secure data platform that supports enterprise-wide analytics, machine learning, and AI initiatives. The Senior Data Engineer will be responsible for designing, building, maintaining and optimising robust data pipelines and enabling governed data sharing with external partners. This role will work closely with data analysts, data scientists, and AI engineers to ensure the scalable, secure, and high-performance data platforms that power analytics, AI, machine learning and operational excellence required by business teams. You will build data assets to provide robust, compliant and agile solutions to support in healthcare, retail, and insurance innovations.

WHAT WE'RE LOOKING FOR?

Minimum

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field
• Certification in cloud platform: AWS, Azure Data Engineer (DP-203/DP-700), or Databricks Certified Data Engineer
• 6+ years of experience in data engineering or software development
• 3+ years of hands-on experience with Databricks Intelligence Platform and implementing Medallion Architecture.
• Strong proficiency in Python, SQL, Spark, and cloud-native data services (e.g., Azure Data Lake, AWS S3, GCP BigQuery)
• Familiarity with cloud-native data integration tools (e.g. SAP Datasphere, Snowflake Airflow)
• Experience in delivery and collaboration tools: GitHub, JIRA, Confluence and MIRO
• Proven experience in developing data platforms that support data science, data analytics and AI workloads
• Extensive experience in ETL development and Data Pipelines with CI/CD, as well as data governance, and data quality frameworks
• Solid understanding of data science workflows, model lifecycle, and AI concepts

Preferred Qualifications

• Certification in Machine Learning or AI (e.g., Azure AI Engineer Associate)
• Experience in healthcare, retail, or insurance data ecosystems
• Working knowledge of SAP BW, S4 HANA, CRM, ERP, and Data Business Cloud

WHAT YOU WILL BE DOING?

• Lead the development and maintenance of a scalable and secure data platform using the Medallion Architecture to support analytics, data science, and AI workloads with high performance and reliability
• Build robust, reusable, and modular data pipelines leveraging Python, SQL, Spark, and cloud-native services to ensure consistent, timely, and accurate data delivery across the organisation
• Integrate diverse data sources and systems using scalable ingestion frameworks, defined patterns and APIs to unify enterprise data and reduce silos
• Collaborate with data analysts, data scientists, AI engineers and data quality analysts through regular engagement, feedback sessions, and shared development environments to ensure the platform meets analytical and modelling needs effectively
• Ensure data integrity, privacy, and compliance by implementing automated data quality checks, lineage
• Document and maintain technical architecture, data flows, integration patterns and engineering standards using version-controlled repositories and collaborative tools to support maintainability, onboarding, and cross-functional transparency

WHO YOU ARE?

• You have advanced knowledge of AWS, Databricks, and Apache Spark.
• You have expertise in Medallion Architecture, having implemented Bronze, Silver, and Gold layers.
• You adapt with confidence in a fast-paced, evolving environment.
• You mentor business users, data professionals, and the data engineering team.
• You deliver work using agile principles and collaboration tools such as GitHub, JIRA, Confluence, and Miro.

Our values show up in how we think, build, and make choices that make an impact. We are building a culture that drives progress; one that fixes friction, chases what changes, and owns the outcome. We design with tomorrow in mind, ask sharper questions, and answer them with care, urgency, and systems that scale.

Think you’ve got energy, curiosity, and guts? Come hold the space, make it matter, and own a breakthrough.