Senior Data Engineer (Big Data & Analytics)

Placements24; Klerksdorp; Hybrid; ZAR 900,000 - 1,500,000; Full time

Benefits offered by this job

Fully remote work

Flexible hours

Home office setup allowance

Professional development

Conferences

Generous paid time off

Retirement fund participation

Job summary

Placements24 is seeking a Senior Data Engineer in South Africa with a strong background in big data and analytics. This remote-first role involves designing, building, and maintaining scalable data pipelines and data lakes to support advanced analytics and decision-making.

You will work with Spark, Hadoop, and cloud platforms to ensure data quality, security, and performance, collaborating with data scientists and business stakeholders in a distributed team environment.

Qualifications

• Minimum of 5 years of data engineering experience with big data technologies.
• Proficiency in Python, Scala or Java.
• Hands-on experience with distributed data processing frameworks like Spark and Hadoop ecosystem.
• Experience with cloud data platforms such as AWS, Azure or GCP.
• Strong SQL skills and experience with relational and NoSQL databases.
• Understanding of data warehousing concepts and ETL/ELT tools.

Responsibilities

• Design, develop, and optimize scalable data pipelines for ingesting, processing, and transforming large volumes of data.
• Build and maintain data warehousing solutions and data lakes using big data technologies.
• Implement data quality frameworks and ensure accuracy, consistency, and reliability of data.
• Develop ETL/ELT processes to support BI and analytics requirements.
• Collaborate with data scientists, analysts, and stakeholders to understand data needs and deliver solutions.
• Monitor and troubleshoot data pipeline performance and data infrastructure issues.
• Stay updated with emerging big data technologies and best practices.

Skills

Python

Scala

Java

SQL

ETL/ELT

Data Modeling

Education

Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or related field

Tools

Apache Spark

Hadoop

HDFS

Kafka

AWS EMR

S3

Redshift

Azure Data Lake

Databricks

GCP BigQuery

Job description

About the Role Our client, a leading telecommunications innovator in Klerksdorp, North West , is seeking a Senior Data Engineer specializing in Big Data and Analytics. This is a remote-first position, offering the flexibility to work from anywhere within South Africa . You will be responsible for designing, building, and maintaining robust data pipelines and infrastructure that support our advanced analytics capabilities. This role is critical in transforming raw data into actionable insights, enabling data-driven decision-making across the organization. You will work with massive datasets, leveraging cutting-edge big data technologies and cloud platforms. The ideal candidate possesses a strong technical background in data engineering, a passion for solving complex data challenges, and the ability to collaborate effectively in a remote team environment. This is an exciting opportunity to shape the data landscape of a dynamic telecommunications company and contribute to its strategic growth through powerful data insights. We foster an environment of continuous learning and innovation for our remote workforce.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
• Build and maintain data warehousing solutions and data lakes using big data technologies (e.g., Hadoop, Spark, Kafka).
• Implement data quality frameworks and ensure the accuracy, consistency, and reliability of data.
• Develop ETL/ELT processes and solutions to support business intelligence and analytics requirements.
• Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver solutions.
• Monitor and troubleshoot data pipeline performance and data infrastructure issues.
• Optimize data storage and processing for cost-efficiency and performance.
• Stay abreast of emerging big data technologies and best practices, recommending and implementing new tools and techniques.
• Ensure data security and compliance with relevant regulations.

Requirements

• Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.
• Minimum of 5 years of experience in data engineering, with a strong focus on big data technologies.
• Proficiency in programming languages such as Python, Scala, or Java.
• Hands-on experience with distributed data processing frameworks like Apache Spark, Hadoop ecosystem (HDFS, MapReduce, Hive).
• Experience with cloud data platforms (e.g., AWS EMR, S3, Redshift; Azure Data Lake, Databricks; GCP BigQuery).
• Strong SQL skills and experience with various database technologies (relational and NoSQL).
• Familiarity with data warehousing concepts and ETL/ELT tools.
• Understanding of data modeling techniques and best practices.
• Excellent problem-solving skills and ability to work independently.
• Strong communication skills for collaborating with technical and non-technical teams.

Benefits

• Competitive salary and comprehensive benefits package.
• Fully remote work arrangement with flexible hours.
• Allowance for home office setup.
• Opportunities for professional development, training, and conferences.
• Cutting-edge technology stack and challenging projects.
• Collaborative and innovative team culture.
• Generous paid time off.
• Retirement fund participation.