Lead Big Data Engineer - Hybrid, Data Lakes & Pipelines

Placements24; Upington; Hybrid; ZAR 1,200,000 - 1,800,000; Full time

Benefits offered by this job

Health benefits

Hybrid work model

Performance bonuses

Career growth opportunities

Job summary

Placements24 in Upington, Northern Cape, seeks a Lead Data Engineer to steer our Big Data Solutions initiatives, designing robust pipelines and infrastructure to power analytics and ML. You will lead a team of data engineers, define best practices, and ensure reliable data flow across the organization.

This hybrid role blends on-site collaboration with remote flexibility in Upington and surrounding regions.

Qualifications

• Master's or Bachelor's degree in Computer Science, Engineering, or related field.
• 7+ years in data engineering with at least 2 years in lead roles.
• Deep experience with Hadoop, Spark, Kafka and Flink.
• Proficiency in Python, Scala, or Java for data processing.
• Strong SQL skills; experience with relational and NoSQL databases.
• Cloud experience across AWS, Azure or GCP with big data services.
• Data warehousing concepts and dimensional modeling.
• Data pipeline orchestration with Airflow or Luigi.
• Excellent leadership, communication, and problem-solving skills.
• Experience working in Upington or hybrids preferred.

Responsibilities

• Lead design, development, and deployment of scalable big data solutions and data warehouses.
• Build and maintain robust ETL/ELT data pipelines across platforms.
• Oversee data lakes, warehouses, and data governance.
• Collaborate with data scientists, analysts, and stakeholders to define data needs.
• Mentor a team of data engineers and drive technical excellence.
• Ensure data quality, security, and compliant data handling.
• Optimize performance and data retrieval efficiency.
• Evaluate and adopt new big data technologies.
• Guide architecture decisions and promote best practices.
• Resolve complex data pipeline and infrastructure issues.

Skills

Leadership

Big data

Hadoop

Spark

Kafka

Flink

Python

Scala

Java

SQL

AWS

Azure

GCP

Airflow

Luigi

Data governance

Data security

Education

Master's/Bachelor's in CS/Engineering

Tools

Hadoop

Spark

Kafka

Flink

Airflow

Luigi

Job description

Placements24 in Upington, Northern Cape, seeks a Lead Data Engineer to steer our Big Data Solutions initiatives, designing robust pipelines and infrastructure to power analytics and ML. You will lead a team of data engineers, define best practices, and ensure reliable data flow across the organization.

This hybrid role blends on-site collaboration with remote flexibility in Upington and surrounding regions.