Lead Data Engineer - Big Data Solutions

Placements24; Upington; Hybrid; ZAR 1,200,000 - 1,800,000; Full time

Benefits offered by this job

Health benefits

Hybrid work model

Performance bonuses

Career growth opportunities

Job summary

Placements24 in Upington, Northern Cape, seeks a Lead Data Engineer to steer our Big Data Solutions initiatives, designing robust pipelines and infrastructure to power analytics and ML. You will lead a team of data engineers, define best practices, and ensure reliable data flow across the organization.

This hybrid role blends on-site collaboration with remote flexibility in Upington and surrounding regions.

Qualifications

• Master's or Bachelor's degree in Computer Science, Engineering, or related field.
• 7+ years in data engineering with at least 2 years in lead roles.
• Deep experience with Hadoop, Spark, Kafka and Flink.
• Proficiency in Python, Scala, or Java for data processing.
• Strong SQL skills; experience with relational and NoSQL databases.
• Cloud experience across AWS, Azure or GCP with big data services.
• Data warehousing concepts and dimensional modeling.
• Data pipeline orchestration with Airflow or Luigi.
• Excellent leadership, communication, and problem-solving skills.
• Experience working in Upington or hybrids preferred.

Responsibilities

• Lead design, development, and deployment of scalable big data solutions and data warehouses.
• Build and maintain robust ETL/ELT data pipelines across platforms.
• Oversee data lakes, warehouses, and data governance.
• Collaborate with data scientists, analysts, and stakeholders to define data needs.
• Mentor a team of data engineers and drive technical excellence.
• Ensure data quality, security, and compliant data handling.
• Optimize performance and data retrieval efficiency.
• Evaluate and adopt new big data technologies.
• Guide architecture decisions and promote best practices.
• Resolve complex data pipeline and infrastructure issues.

Skills

Leadership

Big data

Hadoop

Spark

Kafka

Flink

Python

Scala

Java

SQL

AWS

Azure

GCP

Airflow

Luigi

Data governance

Data security

Education

Master's/Bachelor's in CS/Engineering

Tools

Hadoop

Spark

Kafka

Flink

Airflow

Luigi

Job description

About the Role

Our client, a leader in data analytics and business intelligence, is seeking a visionary Lead Data Engineer to spearhead their Big Data Solutions initiatives. This pivotal role is based in Upington, Northern Cape , and involves designing, building, and maintaining robust data pipelines and infrastructure to support advanced analytics and machine learning capabilities. You will lead a team of data engineers, define best practices, and ensure the efficient and reliable flow of data across the organization. The position offers a hybrid work arrangement, combining on-site collaboration with remote flexibility. We are looking for an experienced leader passionate about leveraging data to drive business insights and innovation.

Key Responsibilities

• Lead the design, development, and implementation of scalable big data solutions and data warehousing systems.
• Build and maintain efficient, reliable, and robust ETL/ELT data pipelines using various big data technologies.
• Oversee the architecture and implementation of data lakes and data warehouses.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.
• Mentor and guide a team of data engineers, fostering technical excellence and professional development.
• Ensure data quality, integrity, and security across all data platforms.
• Optimize data processing performance and data retrieval efficiency.
• Evaluate and adopt new big data technologies and tools.
• Develop and enforce data governance policies and best practices.
• Troubleshoot and resolve complex data pipeline and infrastructure issues.

Requirements

• Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 7+ years of experience in data engineering, with at least 2 years in a lead or senior capacity.
• Extensive experience with big data technologies such as Hadoop, Spark, Kafka, and Flink.
• Proficiency in programming languages like Python, Scala, or Java for data processing.
• Strong SQL skills and experience with relational and NoSQL databases.
• Experience with cloud platforms (AWS, Azure, GCP) and their big data services (e.g., S3, Redshift, BigQuery, Databricks).
• Familiarity with data warehousing concepts and dimensional modeling.
• Experience with data pipeline orchestration tools (e.g., Airflow, Luigi).
• Excellent leadership, communication, and problem-solving skills.
• Experience working in the Upington, Northern Cape region or willingness to work a hybrid schedule is preferred.

Benefits

Our client offers a highly competitive salary, performance-based bonuses, a comprehensive health benefits package, and significant opportunities for professional growth and skill development in the rapidly evolving field of big data. The hybrid work model supports work-life balance. Employees will have the chance to work with cutting-edge technologies, lead impactful data initiatives, and contribute to data-driven decision-making within a forward-thinking organization.