Senior Data Engineer

BETSoftware; Johannesburg; On-site; ZAR 800,000 - 1,200,000; Full time

Job summary

BETSoftware is seeking an experienced Data Engineer to design, build, and enhance batch and real-time data pipelines that power enterprise analytics and operations. You will craft reusable patterns for ingestion, transformation, and serving layers across the platform, ensuring reliability and scalability in production.

The role emphasizes leading technical direction, mentoring junior engineers, and modernising legacy workflows into stronger, platform-aligned solutions while collaborating with BI

Qualifications

• Degree or diploma in IT, Computer Science, Engineering, or a related technical discipline.
• 6+ years of experience in data engineering, ETL/ELT development, or data platform engineering.
• Strong hands-on SQL expertise including performance tuning and efficient analytic design.
• Experience building and operating modern data platform components in production environments.

Responsibilities

• Design, build, and improve critical data pipelines supporting analytics and operational use cases.
• Develop reusable patterns for ingestion, transformation, storage, and serving layers across the platform.
• Own platform components ensuring reliability, scalability, and supportability in production.
• Mentor junior data engineers through code reviews and knowledge sharing.

Skills

SQL proficiency

Data modeling

Performance tuning

Problem solving

Education

Degree or diploma in IT/CS/Engineering

Tools

Airflow

Spark

Flink

Kafka

ClickHouse

Job description

Responsibilities

• Design, build, and improve critical batch and near‑real‑time data pipelines that support enterprise analytics and operational use cases.
• Develop reusable engineering patterns for ingestion, transformation, storage, and serving layers across the platform.
• Own important platform components and ensure they are reliable, scalable, and supportable in production.
• Contribute directly to the modernisation of legacy data workflows into stronger platform‑aligned solutions.

Lakehouse Architecture & Scalable Processing

• Contribute to the design and evolution of the enterprise data lake or lakehouse platform.
• Implement and refine engineering standards for storage layout, transformation patterns, and data processing frameworks.
• Optimise partitioning, schema evolution, and file organisation to improve performance and maintainability.
• Build and support distributed data processing solutions using modern frameworks and platform tooling.

Data Quality, Reliability & Governance

• Design and implement data quality, reconciliation, and observability controls for critical datasets and platform flows.
• Ensure key pipelines and datasets meet expectations for freshness, completeness, accuracy, and recoverability.
• Strengthen metadata, lineage, and documentation practices across the platform.
• Work with governance and security stakeholders to support compliant, well‑controlled data management practices.

Technical Leadership & Collaboration

• Partner with BI, analytics, software engineering, product, and business stakeholders to understand and support data use cases.
• Translate business and platform requirements into scalable, well‑engineered technical solutions.
• Provide hands‑on technical leadership during planning, design, implementation, and operational improvement.
• Mentor intermediate and junior data engineers through code reviews, design guidance, and practical knowledge‑sharing.

Continuous Improvement & Innovation

• Identify opportunities to improve platform performance, resilience, scalability, and cost efficiency.
• Drive automation, standardisation, and maintainability across the data platform.
• Evaluate new tools or patterns where they provide clear value to the platform or team.
• Help grow the maturity of BET Software's data engineering capability over time.

Tech Environment

• The platform may include a combination of established and modern technologies such as: SQL Server, Python, Spark, Flink, Airflow, Object storage, Open format Tables, Kafka / Redpanda, ClickHouse or similar columnar analytical stores, Git and CI/CD tooling, Kubernetes, OpenShift, or similar runtime environments.

Qualifications

• Degree or diploma in IT, Computer Science, Engineering, or a related technical discipline.
• 6+ years of experience in data engineering, ETL/ELT development, or data platform engineering.
• Strong hands‑on SQL expertise, including advanced performance tuning, query optimisation, indexing strategies, and efficient analytical data design.
• Proven experience building and operating modern data platform components in production environments.
• Strong experience working with object storage in cloud or on‑premises environments.
• Experience with workflow orchestration platforms such as Airflow, SQL Server Agent, or similar.

Technical & Architectural Skills

• Data Warehouse, Lake and Lakehouse architecture patterns.
• Distributed data processing using frameworks such as Spark or Flink.
• Designing and supporting batch and near‑real‑time ingestion pipelines.
• Building incremental, idempotent, and fault‑tolerant data pipelines.
• Data quality, reconciliation, and observability practices.
• Metadata, lineage, governance, and access control concepts.
• Analytical data modelling and efficient data structures for warehouse and large‑scale query workloads.

Experience In The Following Areas Is Highly Valuable

• Medallion architecture such as Bronze, Silver, and Gold layers.
• Open table formats such as Iceberg.
• Schema evolution, partitioning strategies, file optimisation, and storage layout tuning.
• Event‑driven or streaming platforms such as Kafka, Pulsar, or Redpanda.
• Columnar or high‑performance analytical platforms such as ClickHouse.
• CI/CD pipelines, deployment automation, and engineering standards for data workloads.

Platform & Engineering Practices

• Experience improving reliability, performance, and scalability across production data platforms.
• Familiarity with monitoring, alerting, observability, and operational support practices.
• Exposure to containerised or clustered environments such as Kubernetes, OpenShift, or similar platforms.
• Strong debugging capability across data, pipeline, compute, and platform layers.

Personal Attributes

• Strong sense of ownership and accountability.
• Comfortable making technical trade‑offs while remaining pragmatic and hands‑on.
• Excellent problem‑solving and communication skills.
• Able to collaborate effectively with technical and non‑technical stakeholders.
• Passionate about building scalable, maintainable, and well‑documented systems.
• Committed to sharing knowledge and raising engineering standards across the team.

Living the Spirit

• Engages in cross‑functional collaboration and problem solving while contributing to an inclusive team culture.
• Supports a culture of adaptability, accountability, and shared success across the department and wider business.
• Shows up authentically and contributes to team outcomes by working effectively with diverse colleagues and perspectives.
• Approaches challenges as opportunities to learn, improve, and help others grow.