Data Pipeline Engineer: Batch & Real‑Time Analytics

BETSoftware; Johannesburg; On-site; ZAR 520,000 - 900,000; Full time

Job summary

BETSoftware in Johannesburg is seeking a data engineer to build, maintain, and optimize batch and near real-time data pipelines that support analytics and business operations.

You will work across ingestion, transformation, storage, and serving layers, using SQL, Python, and related tooling, while modernising legacy workflows into scalable, reliable patterns.

Collaborate with BI, analytics, and product teams to ensure data quality and accessibility for stakeholders.

Qualifications

• Degree or diploma in IT, CS, engineering, or a related technical discipline.
• 2–4 years of data engineering, ETL/ELT, or data platform engineering experience.
• Hands-on SQL experience with query optimisation and performance tuning.
• Experience building and maintaining batch data pipelines in production.
• Exposure to data warehouse, data lake concepts, or object storage.
• Experience with orchestration platforms such as Airflow, Dagster, or similar.

Responsibilities

• Build, maintain, and improve batch and near real-time data pipelines.
• Contribute to ingestion, transformation, storage, and serving layers within the data platform.
• Develop reliable data processing logic using SQL, Python, and related tooling.
• Modernise legacy data workflows into scalable, maintainable patterns.
• Contribute to data lake or lakehouse components.
• Prepare data for analytics across warehouse and large-storage environments.
• Assist with data layout, partitioning, and optimisation practices.
• Work with senior engineers to apply platform standards and architectural patterns.
• Build and maintain data quality checks, reconciliation logic, and pipeline validation.
• Ensure datasets meet freshness, completeness, and accuracy expectations.
• Support metadata, lineage, and documentation practices.
• Troubleshoot pipeline failures, data discrepancies, and performance issues.
• Collaborate with BI, analytics, software eng, product, and business stakeholders.
• Translate requirements into scalable, well-engineered solutions.
• Participate in planning, estimation, implementation, testing, and deployment.

Skills

SQL

Data pipelines

ETL/ELT

Airflow/Dagster

Education

Degree or diploma in IT/CS/Engineering

Tools

SQL Server

Python

Spark

Flink

Airflow

Dagster

Kafka

Kubernetes

Job description

BETSoftware in Johannesburg is seeking a data engineer to build, maintain, and optimize batch and near real-time data pipelines that support analytics and business operations.

You will work across ingestion, transformation, storage, and serving layers, using SQL, Python, and related tooling, while modernising legacy workflows into scalable, reliable patterns.

Collaborate with BI, analytics, and product teams to ensure data quality and accessibility for stakeholders.