Senior Data Engineer - Big Data

Placements24; Sol Plaatje Local Municipality; Hybrid; ZAR 1,000,000 - 1,400,000; Full time

Benefits offered by this job

Competitive salary with performanceBon

Health, dental, and vision insurance

Professional development and training

Hybrid work model in Upington

Generous paid time off

Company holidays

Job summary

Placements24 client in Upington is seeking a Senior Data Engineer specializing in Big Data to design, build and maintain robust data pipelines and infrastructure supporting analytics and BI. You will handle large, complex datasets and ensure data quality, availability and accessibility for analysis.

This hybrid role offers collaboration, cutting-edge data solutions, and opportunities to implement scalable data warehouses, lakes, and streaming pipelines with Spark, Hadoop and Kafka.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related quantitative field, or equivalent experience.
• 5+ years of experience in data engineering, with a strong focus on big data technologies and distributed systems.
• Proficiency with programming languages such as Python, Scala, or Java.
• Hands-on experience with big data processing frameworks (e.g., Apache Spark, Hadoop MapReduce).
• Experience with cloud-based data platforms (e.g., AWS EMR, Azure Databricks, Google Dataflow).
• Strong understanding of SQL, NoSQL databases, and data warehousing concepts.
• Experience with stream processing technologies (e.g., Kafka, Kinesis) is a plus.

Responsibilities

• Design, develop, and maintain scalable and efficient big data pipelines using Spark, Hadoop, Kafka, and related ecosystem tools.
• Build and manage data warehousing solutions and data lakes to support business analytics and reporting needs.
• Implement data quality checks, monitoring, and validation processes to ensure data integrity.
• Collaborate with data scientists and analysts to understand data requirements and provide clean, reliable data sets.
• Optimize data processing performance and database queries for speed and efficiency.
• Develop and maintain ETL/ELT processes for integrating data from various sources.
• Ensure data security, privacy, and compliance with relevant regulations.

Skills

Python

Scala

Java

Spark

Hadoop

Kafka

SQL

NoSQL

Data warehousing

ETL/ELT

Data quality

Data security

Education

Bachelor's degree in Computer Science, Engineering, or related quantitative field

Tools

Apache Spark

Hadoop MapReduce

AWS EMR

Azure Databricks

Google Dataflow

Kafka

Job description

About the Role

Our client is seeking an accomplished Senior Data Engineer with a specialization in Big Data technologies to join their team in Upington . This role is essential for designing, building, and maintaining robust data pipelines and infrastructure that support analytics and business intelligence initiatives. You will work with large, complex datasets, implementing solutions that ensure data quality, availability, and accessibility for analysis. This position offers a challenging and rewarding opportunity to work on cutting-edge data solutions within a hybrid work environment that values collaboration and innovation.

Key Responsibilities

• Design, develop, and maintain scalable and efficient big data pipelines using technologies like Spark, Hadoop, Kafka, and related ecosystem tools.
• Build and manage data warehousing solutions and data lakes to support business analytics and reporting needs.
• Implement data quality checks, monitoring, and validation processes to ensure data integrity.
• Collaborate with data scientists and analysts to understand data requirements and provide clean, reliable data sets.
• Optimize data processing performance and database queries for speed and efficiency.
• Develop and maintain ETL/ELT processes for integrating data from various sources.
• Ensure data security, privacy, and compliance with relevant regulations.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field, or equivalent experience.
• 5+ years of experience in data engineering, with a strong focus on big data technologies and distributed systems.
• Proficiency with programming languages such as Python, Scala, or Java.
• Hands-on experience with big data processing frameworks (e.g., Apache Spark, Hadoop MapReduce).
• Experience with cloud-based data platforms (e.g., AWS EMR, Azure Databricks, Google Dataflow).
• Strong understanding of SQL, NoSQL databases, and data warehousing concepts.
• Experience with stream processing technologies (e.g., Kafka, Kinesis) is a plus.

Benefits

• Competitive annual salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Opportunities for professional development, training, and certifications.
• Hybrid work model offering flexibility and in-office collaboration in Upington.
• A dynamic and supportive work environment focused on data excellence.
• Generous paid time off and company holidays.