Data Engineer - Big Data Platforms

Placements24; Siyancuma Local Municipality; Hybrid; ZAR 550,000 - 750,000; Full time

Job summary

Placements24 is seeking a Data Engineer for Upington to design, build and maintain scalable data pipelines and platforms, enabling efficient data processing, storage and analysis. You will ensure data availability and integrity to support data scientists and analysts.

You will work with big data technologies, develop data warehouses and data lakes, monitor performance, and collaborate with stakeholders to deliver data solutions.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 3+ years of experience as a Data Engineer or in a similar role involving big data technologies.
• Proficiency with distributed data processing frameworks such as Spark, Hadoop, or Flink.
• Experience with SQL and NoSQL databases, as well as data warehousing concepts.
• Knowledge of cloud platforms (AWS, Azure, GCP) and their data services is a plus.
• Strong programming skills in Python, Scala, or Java.

Responsibilities

• Design, build, and optimize robust and scalable data pipelines for ingesting, transforming, and processing large volumes of data.
• Develop and maintain data warehousing solutions and data lakes using modern big data technologies.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Monitor data infrastructure performance, troubleshoot issues, and implement optimizations.
• Stay current with emerging big data technologies and best practices.

Skills

Python

Scala

Java

SQL

NoSQL

Spark

Hadoop

Cloud platforms

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

AWS

Azure

GCP

Job description

About the Role

Our client is seeking a dedicated Data Engineer to build and manage their big data infrastructure in Upington . You will be responsible for designing, constructing, and maintaining scalable data pipelines and platforms that enable efficient data processing, storage, and analysis. This role is crucial for ensuring the availability and integrity of data assets, supporting data scientists and analysts in their work. If you have a passion for working with large-scale data systems and driving data-driven insights, this is an excellent opportunity.

Key Responsibilities

• Design, build, and optimize robust and scalable data pipelines for ingesting, transforming, and processing large volumes of data.


• Develop and maintain data warehousing solutions and data lakes using modern big data technologies.


• Ensure data quality, integrity, and security across all data platforms.


• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.


• Monitor data infrastructure performance, troubleshoot issues, and implement optimizations.


• Stay current with emerging big data technologies and best practices.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field.


• 3+ years of experience as a Data Engineer or in a similar role involving big data technologies.


• Proficiency with distributed data processing frameworks such as Spark, Hadoop, or Flink.


• Experience with SQL and NoSQL databases, as well as data warehousing concepts.


• Knowledge of cloud platforms (AWS, Azure, GCP) and their data services is a plus.


• Strong programming skills in Python, Scala, or Java.

Benefits

• Competitive salary and performance-based bonuses.


• Comprehensive health and retirement benefits.


• Opportunities for training and professional development in data engineering.


• A collaborative environment focused on data innovation in Upington .


• Contribution to building and maintaining critical data infrastructure.