Senior Data Platform Engineer: Medallion Architecture & AI

X, bigly labs; Johannesburg; On-site; ZAR 1,200,000 - 1,800,000; Full time

Job summary

X, bigly labs in Johannesburg seeks a Senior Data Engineer to lead the development of a modern, scalable, and secure data platform that powers analytics, machine learning, and AI initiatives. You will design, build, and optimise robust data pipelines to enable governed data sharing with external partners.

You will collaborate with data analysts, data scientists, and AI engineers to ensure high-performance platforms supporting healthcare, retail, and insurance innovations.

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Certification in cloud platform: AWS, Azure Data Engineer, or Databricks Certified Data Engineer.
• 6+ years of experience in data engineering or software development.
• 3+ years with Databricks Intelligence Platform and Medallion Architecture.
• Strong proficiency in Python, SQL, Spark, and cloud-native data services.
• Familiarity with cloud-native data integration tools (e.g. SAP Datasphere, Snowflake Airflow).
• Experience with GitHub, JIRA, Confluence and MIRO.
• Proven experience in data platforms for data science, analytics, and AI workloads.
• ETL development with CI/CD, data governance, and data quality frameworks.
• Solid understanding of data science workflows and AI concepts.

Responsibilities

• Lead development of a scalable and secure data platform using Medallion Architecture for analytics, data science, and AI workloads.
• Build robust, modular data pipelines with Python, SQL, Spark, and cloud-native services.
• Integrate diverse data sources with scalable ingestion frameworks and APIs.
• Collaborate with analysts, data scientists, AI engineers, and data quality analysts.
• Ensure data integrity, privacy, and compliance with automated data quality checks and lineage.
• Document architecture, data flows, and standards in version-controlled repos.

Skills

Python

SQL

Spark

Cloud platforms

Databricks

Medallion Architecture

CI/CD for ETL

Data governance

Data integration tools

GitHub/JIRA/Confluence/MIRO

Education

Bachelor’s degree in Computer Science or related

Tools

Databricks

Apache Spark

AWS

Azure

GCP BigQuery

Snowflake

Airflow

CI/CD tooling

Job description

X, bigly labs in Johannesburg seeks a Senior Data Engineer to lead the development of a modern, scalable, and secure data platform that powers analytics, machine learning, and AI initiatives. You will design, build, and optimise robust data pipelines to enable governed data sharing with external partners.

You will collaborate with data analysts, data scientists, and AI engineers to ensure high-performance platforms supporting healthcare, retail, and insurance innovations.