Intermediate Data Engineer

BETSoftware; Johannesburg; On-site; ZAR 520,000 - 900,000; Full time

Job summary

BETSoftware in Johannesburg is seeking a data engineer to build, maintain, and optimize batch and near real-time data pipelines that support analytics and business operations.

You will work across ingestion, transformation, storage, and serving layers, using SQL, Python, and related tooling, while modernising legacy workflows into scalable, reliable patterns.

Collaborate with BI, analytics, and product teams to ensure data quality and accessibility for stakeholders.

Qualifications

• Degree or diploma in IT, CS, engineering, or a related technical discipline.
• 2–4 years of data engineering, ETL/ELT, or data platform engineering experience.
• Hands-on SQL experience with query optimisation and performance tuning.
• Experience building and maintaining batch data pipelines in production.
• Exposure to data warehouse, data lake concepts, or object storage.
• Experience with orchestration platforms such as Airflow, Dagster, or similar.

Responsibilities

• Build, maintain, and improve batch and near real-time data pipelines.
• Contribute to ingestion, transformation, storage, and serving layers within the data platform.
• Develop reliable data processing logic using SQL, Python, and related tooling.
• Modernise legacy data workflows into scalable, maintainable patterns.
• Contribute to data lake or lakehouse components.
• Prepare data for analytics across warehouse and large-storage environments.
• Assist with data layout, partitioning, and optimisation practices.
• Work with senior engineers to apply platform standards and architectural patterns.
• Build and maintain data quality checks, reconciliation logic, and pipeline validation.
• Ensure datasets meet freshness, completeness, and accuracy expectations.
• Support metadata, lineage, and documentation practices.
• Troubleshoot pipeline failures, data discrepancies, and performance issues.
• Collaborate with BI, analytics, software eng, product, and business stakeholders.
• Translate requirements into scalable, well-engineered solutions.
• Participate in planning, estimation, implementation, testing, and deployment.

Skills

SQL

Data pipelines

ETL/ELT

Airflow/Dagster

Education

Degree or diploma in IT/CS/Engineering

Tools

SQL Server

Python

Spark

Flink

Airflow

Dagster

Kafka

Kubernetes

Job description

Job Description

Build, maintain, and improve batch and near‑real‑time data pipelines that support analytics and operational use cases.

Contribute to ingestion, transformation, storage, and serving layers within the enterprise data platform.

Develop reliable and maintainable data processing logic using SQL, Python, and related tooling.

Help modernise legacy data workflows into more scalable and supportable patterns.

Responsibilities

• Build, maintain, and improve batch and near‑real‑time data pipelines that support analytics and operational use cases.
• Contribute to ingestion, transformation, storage, and serving layers within the enterprise data platform.
• Develop reliable and maintainable data processing logic using SQL, Python, and related tooling.
• Help modernise legacy data workflows into more scalable and supportable patterns.
• Contribute to the implementation and improvement of data lake or lakehouse components.
• Support the preparation of data for efficient analytical use across warehouse and large‑scale storage environments.
• Assist with data layout, partitioning, and optimisation practices that improve platform performance and maintainability.
• Work with senior engineers to apply platform standards and architectural patterns consistently.
• Build and maintain data quality checks, reconciliation logic, and pipeline validation routines.
• Help ensure datasets meet expectations for freshness, completeness, and accuracy.
• Support metadata, lineage, and documentation practices that improve trust and transparency.
• Troubleshoot pipeline failures, data discrepancies, and performance issues in production.
• Partner with BI, analytics, software engineering, product, and business stakeholders to understand and support data use cases.
• Translate business and platform requirements into scalable, well‑engineered technical solutions.
• Participate in planning, estimation, implementation, testing, and deployment activities.
• Contribute to code reviews, documentation, and shared engineering standards.
• Identify opportunities to improve automation, reliability, scalability, and maintainability across the platform.
• Stay current with emerging data engineering practices, tools, and patterns.
• Grow technical capability in modern data platform engineering through hands‑on project delivery.

Tech Environment

The platform may include a combination of established and modern technologies such as SQL Server, Python, Spark, Flink, Airflow, Object storage, Open format Tables, Kafka or Redpanda, ClickHouse or similar columnar analytical stores, Git and CI/CD tooling, Kubernetes, OpenShift, or similar runtime environments.

Qualifications

• Degree or diploma in IT, Computer Science, Engineering, or a related technical discipline.
• 2 to 4 years of experience in data engineering, ETL/ELT development, or data platform engineering.
• Hands‑on SQL experience, including query optimisation, indexing, and performance tuning.
• Experience building and maintaining batch data pipelines in production environments.
• Exposure to data platform concepts such as data warehouse, data lake architecture, or object storage.
• Experience with orchestration platforms such as Airflow, Dagster, or similar.

Living the Spirit

• Engages in cross‑functional collaboration and problem solving while contributing to an inclusive team culture.
• Supports a culture of adaptability and shared accountability across the department and wider business.
• Shows up authentically and contributes to team success by working effectively with diverse colleagues and perspectives.
• Approaches challenges as opportunities to learn, improve, and help others grow.