Senior Data Engineer - Big Data & AI

Placements24; Klerksdorp; Hybrid; ZAR 900,000 - 1,500,000; Full time

Benefits offered by this job

Fully remote position

Health and wellness benefits

Salary and bonus structure

Continuous learning opportunities

Collaborative and innovative team

Job summary

Placements24 in Klerksdorp is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and data infrastructure for analytics and AI initiatives. You will ensure data quality, governance, and accessibility for data scientists and analysts across big data platforms.

The role requires strong SQL and programming skills (Python/Scala), 5+ years in data engineering with Spark/Hadoop/Kafka, and experience with cloud data services.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of data engineering experience with big data tech (Spark, Hadoop, Kafka).
• Proficient in SQL and programming in Python or Scala.
• Experience with cloud data platforms and data services.
• Knowledge of data modelling, database design, and warehousing concepts.
• Familiarity with data pipeline orchestration tools (Airflow) and MLOps is a plus.

Responsibilities

• Design, build, and maintain robust ETL/ELT pipelines for large datasets.
• Develop and manage data warehousing and data lakes using big data tech.
• Ensure data quality, integrity, and security across data systems.
• Collaborate with data scientists and analysts to provide data access.
• Optimize data infrastructure for performance, cost-efficiency, and reliability.
• Implement data governance and monitoring solutions.

Skills

Data engineering

Spark

Hadoop

Kafka

SQL

Python

Scala

Airflow

Cloud data platforms

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

Cloud platforms (AWS/Azure/GCP)

Job description

About the Role

Our client is seeking an experienced Senior Data Engineer to build and maintain scalable data infrastructure that supports advanced analytics and AI initiatives in Klerksdorp . You will be responsible for designing, developing, and optimizing data pipelines, ensuring the availability and quality of data for machine learning models and business intelligence. This role is critical for empowering data scientists and analysts with the data they need to drive insights and innovation. The ideal candidate has a strong background in big data technologies and a passion for enabling data-driven decision-making.

Key Responsibilities

• Design, build, and maintain robust, scalable ETL/ELT pipelines for large datasets.
• Develop and manage data warehousing solutions and data lakes using big data technologies.
• Ensure data quality, integrity, and security across all data systems.
• Collaborate with data scientists and analysts to understand data requirements and provide necessary data access.
• Optimize data infrastructure for performance, cost-efficiency, and reliability.
• Implement data governance best practices and monitoring solutions.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering, with a focus on big data technologies (e.g., Spark, Hadoop, Kafka).
• Proficiency in SQL and programming languages like Python or Scala.
• Experience with cloud data platforms (AWS, Azure, GCP) and their data services.
• Familiarity with data modeling, database design, and data warehousing concepts.
• Strong understanding of data pipeline orchestration tools (e.g., Airflow) and MLOps concepts is a plus.

Benefits

• Attractive salary and bonus structure.
• Fully remote position, providing exceptional work-life balance.
• Comprehensive health and wellness benefits package.
• Opportunities for continuous learning and career advancement in a growing field.
• Be part of a collaborative and innovative team focused on cutting-edge data solutions.