Intermediate Data Engineer

BETSoftware; Wes-Kaap; On-site; ZAR 600 000 - 800 000; Full time

Job summary

A leading software provider in South Africa is seeking a skilled Data Engineer to design and manage high-throughput data pipelines and analyze complex datasets. The ideal candidate will have at least 3 years of experience in data engineering, proficiency in SQL and programming languages like Python or Java, and a strong understanding of big data technologies. We encourage a diverse and collaborative environment where innovation thrives.

Benefits

Collaborative work environment

Professional growth opportunities

Qualifications

• At least 3 years in a technical role with experience in data warehousing and data engineering.
• Proficiency in programming languages such as Python, Java, or Scala for data processing.
• Experience with big data technologies such as Hadoop, Spark, Hive, and Airflow will be advantageous.

Responsibilities

• Design and manage high-throughput, low-latency data pipelines using distributed computing frameworks.
• Build scalable ETL/ELT workflows using tools like Airflow and Spark.
• Ensure efficient data ingestion, transformation, and integration from multiple sources.

Skills

SQL

Python

Java

Scala

Hadoop

Machine Learning

Data Engineering

Data Warehousing

Analytical

Education

Degree or diploma in IT or related technical discipline

Tools

PostgreSQL

MySQL

SQL Server

Hadoop

Spark

Airflow

Job description

Job Description

• SQL
• Hadoop
• SQL MS
• Data Engineering
• Data Warehousing
• Python
• Java
• Scala
• Analytical
• Machine Learning

Responsibilities

Data Engineering

• Design and manage high-throughput, low-latency data pipelines using distributed computing frameworks.
• Build scalable ETL/ELT workflows using tools like Airflow and Spark.
• Work with containerised environments (e.g., Kubernetes, OpenShift) and real-time data platforms (e.g., Apache Kafka, Flink).
• Ensure efficient data ingestion, transformation, and integration from multiple sources.
• Maintain data integrity, reliability, and governance across systems.

Data Analysis And Modelling

• Apply statistical and machine learning techniques to analyse data and translate complex data sets to identify patterns, trends and actionable insights that drive business strategy and operational efficiency.
• Develop predictive models, recommendation systems, and optimisation algorithms to solve business challenges and enhance operational efficiency.
• Transform raw data into meaningful features that improve model performance and translate business challenges into analytical problems providing data driven solutions.

Design and Planning Data Engineering Solutions

• Design and implement testing frameworks to measure the impact of business interventions.
• Design and implement scalable, high-performance big data applications that support analytical and operational workloads.
• Assist with evaluations and recommend best-fit technologies for real-time and batch data processing.
• Ensure that data solutions are optimised for performance, security, and scalability.
• Develop and maintain data models, schemas, and architecture blueprints for relational and big data environments.
• Ensure seamless data integration from multiple sources, leveraging Kafka for real-time streaming and event-driven architecture.
• Facilitate system design and review, ensuring compatibility with existing and future systems.
• Optimise data workflows, ETL/ELT pipelines, and distributed storage strategies.

Technical Development And Innovation

• Keep abreast of technological advancements in data science, data engineering, machine learning and AI.
• Continuously evaluate and experiment with new tools, libraries, and platforms to ensure that the team is using the most effective technologies.
• Work on end-to-end and data engineering projects that support strategic goals. This includes requirements gathering, technical deliverable planning, output quality and stakeholder management.
• Continuous research on to develop and implement innovative ideas and improved methods, systems and work processes which lead to higher quality and better results.
• Build and maintain Kafka-based streaming applications for real-time data ingestion, processing, and analytics.
• Design and implementation data lake and data warehouse data processing & ingestion applications.
• Utilise advanced SQL\Spark query optimisation techniques, indexing strategies, partitioning, and materialised views to enhance performance.
• Work extensively with relational databases (PostgreSQL, MySQL, SQL Server) and big data technologies (Hadoop, Spark).
• Design and implement data architectures that efficiently handle structured and unstructured data at scale.

Resourceful And Improving

• Find innovative ways following processes to overcome challenges, leveraging available tools, data, and methodologies effectively.
• Continuously seek out new techniques, best practices and emerging trends in Data Science, AI, and machine learning.
• Actively contribute to team learning by sharing insights, tools and approaches that improve overall performance.

Qualifications

• Degree or diploma in IT or related technical discipline.
• At least 3 years in a technical role with experience in data warehousing, and data engineering.
• Proficiency in programming languages such as Python, Java, or Scala for data processing.
• Proficiency in SQL for data processing using SQL MS server or PostgreSQL.
• 2-3 years’ experience across the data science workflow will be advantageous.
• 2-3 years of proven experience as a data scientist, with expertise in machine learning, statistical analysis and data visualisation will be advantageous.
• Experience with big data technologies such as Hadoop, Spark, Hive, and Airflow will be advantageous.
• Expertise in SQL/Spark performance tuning, database optimisation, and complex query development will be advantageous.
• Advantageous on .net Programming (C#, C++, Java) and Design Patterns.

Living Our Spirit

• Adaptability & Resilience: Embrace change with flexibility, positivity, and a proactive mindset. Thrive in dynamic, fast-paced environments by adjusting to evolving priorities and technologies.
• Decision-Making & Accountability: Make timely, data-informed decisions involving the team to ensure transparency and alignment. Confidently justify choices based on thorough analysis and sound judgment.
• Innovation & Continuous Learning: Actively pursue new tools, techniques, and best practices in Data Science, AI, and engineering. Share insights openly to foster team growth and continuously improve performance.
• Collaboration & Inclusion: Foster open communication and create a supportive, inclusive environment where diverse perspectives are valued. Empower team members to share ideas, seek help, and give constructive feedback freely.
• Leadership & Growth: Lead authentically with integrity and openness. Support team members through mentorship, skill development, and creating a safe space for honest feedback and innovation. Celebrate successes and embrace challenges as growth opportunities.

About Us

We Don't Just Recruit Talent... We Cultivate It

BET Software is one of the most diverse betting software providers in Sub-Saharan Africa, with a rapidly growing international footprint. Our multiskilled Team is responsible for providing advanced software solutions, while supporting incredibly high transactional volumes in a fast-paced industry, making BET Software a lively and engaging place to be.

Our progressive use of technology enables our talented Team to work with the tech stack of their choice and we encourage an environment and culture that supports collaboration, learning, and growth. We believe that our strength lies in our diversity and that Teamwork makes the dream work, a place where you can flourish amongst like-minded individuals.

We are a dynamic group of people that stem from a variety of cultures, backgrounds, and locations. While our head office is based in Umhlanga, South Africa we recruit talent throughout South Africa and Internationally.