Senior Data Engineer: Scalable Pipelines & Data Lakes

Placements24; Upington; On-site; ZAR 900,000 - 1,300,000; Full time

Benefits offered by this job

Medical aid

Retirement fund contributions

Training and certifications

Annual performance bonuses

Generous paid time off

Employee assistance programs

Job summary

WhatJobs Direct is seeking a skilled and experienced Senior Data Engineer to join our growing data analytics team in Upington, Northern Cape. This role focuses on designing, building, and maintaining data pipelines and infrastructure to enable efficient data warehousing and analytics for data scientists and analysts.

The ideal candidate will have 5+ years in data engineering, strong programming skills in Python/SQL/Scala, and hands-on experience with Spark, Hadoop, Kafka, and cloud platforms

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related quantitative field, or equivalent work experience.
• Minimum of 5 years of experience in data engineering or a related role.
• Strong programming skills in Python, SQL, or Scala.
• Proficiency with ETL/ELT tools and data integration techniques.
• Experience with relational databases (e.g., PostgreSQL, SQL Server) and NoSQL databases (e.g., MongoDB, Cassandra).
• Hands-on experience with big data technologies such as Apache Spark, Hadoop, or Kafka.
• Experience with cloud data platforms (e.g., AWS Redshift, Azure Data Lake, Google BigQuery) is highly desirable.
• Knowledge of data warehousing concepts and dimensional modeling.
• Strong analytical and problem-solving skills.
• Excellent communication and teamwork abilities.

Responsibilities

• Design, build, and maintain scalable and reliable data pipelines (ETL/ELT processes).
• Develop and manage data warehouses, data lakes, and other data storage solutions.
• Optimize data retrieval and processing performance.
• Ensure data quality, integrity, and security across all data systems.
• Collaborate with data scientists and analysts to understand their data needs and provide efficient data solutions.
• Implement and manage data integration from various sources.
• Automate data processes and workflows.
• Monitor data pipeline performance and troubleshoot issues proactively.
• Stay current with emerging data engineering technologies and best practices.
• Contribute to the design and implementation of the overall data architecture.
• Document data processes, schemas, and architecture.

Skills

Python

SQL

Scala

ETL/ELT concepts

Data integration

Data modeling

Big data concepts

Education

Bachelor's degree in Computer Science, Engineering, Information Technology, or related field

Tools

PostgreSQL

SQL Server

MongoDB

Cassandra

Apache Spark

Hadoop

Kafka

AWS Redshift

Azure Data Lake

Google BigQuery

Job description

WhatJobs Direct is seeking a skilled and experienced Senior Data Engineer to join our growing data analytics team in Upington, Northern Cape. This role focuses on designing, building, and maintaining data pipelines and infrastructure to enable efficient data warehousing and analytics for data scientists and analysts.

The ideal candidate will have 5+ years in data engineering, strong programming skills in Python/SQL/Scala, and hands-on experience with Spark, Hadoop, Kafka, and cloud platforms