Lead Data Engineer

Placements24; Polokwane; Hybrid; ZAR 900,000 - 1,800,000; Full time

Benefits offered by this job

Medical insurance

Life insurance

Retirement fund

Bonuses

Leadership training

Hybrid work Polokwane

Job summary

Placements24 is seeking an experienced Lead Data Engineer for a hybrid role based in Polokwane to design, develop, and maintain scalable data infrastructure for a major e-commerce analytics client.

You will lead a team of data engineers, build robust data pipelines, and collaborate with data scientists and business stakeholders to deliver high-quality analytics, BI, and ML initiatives in a hybrid work setup.

Qualifications

• 7+ years in data engineering, with at least 3 years in a lead/senior capacity.
• Strong proficiency in SQL and experience with relational and NoSQL databases.
• Hands-on experience with Spark, Hadoop, Flink, or similar big data frameworks.
• Expertise building ETL/ELT pipelines using Airflow, Kafka, or equivalent.
• Experience with cloud data platforms (AWS/Azure/GCP).
• Proficiency in Python/Scala/Java for data tooling.
• Understanding of data warehousing concepts and data modeling.
• Experience with data governance and data quality frameworks.

Responsibilities

• Design, build, and maintain scalable data pipelines and ETL/ELT processes.
• Develop and optimize data warehousing solutions and data lakes.
• Oversee big data tech (Spark, Hadoop ecosystem) implementation.
• Ensure data quality, integrity, and consistency across platforms.
• Collaborate with data scientists, analysts, and stakeholders to meet data needs.
• Lead and mentor a team of data engineers, promoting best practices.
• Implement data governance and ensure privacy compliance.
• Automate data processes and monitor performance to resolve bottlenecks.
• Evaluate new data technologies to enhance infrastructure.
• Contribute to the architectural roadmap and document pipelines.

Skills

SQL

Spark

Hadoop

Airflow

Kafka

Python

Scala/Java

Leadership

NoSQL

Data modeling

Education

Master's degree in CS/Engineering

Equivalent experience

Tools

Cloud platforms (AWS/Azure/GCP)

Job description

About the Role

Our client, a major player in the e-commerce analytics space, is searching for an experienced Lead Data Engineer to spearhead the design, development, and maintenance of their data infrastructure. This role is crucial for building scalable and efficient data pipelines that support business intelligence, analytics, and machine learning initiatives. You will lead a team of data engineers, ensuring high standards of data quality, performance, and reliability. This position offers a hybrid work model , combining remote flexibility with essential in-office collaboration in Polokwane .

Key Responsibilities

• Design, build, and maintain robust, scalable, and highly available data pipelines and ETL/ELT processes.
• Develop and optimize data warehousing solutions and data lakes.
• Oversee the implementation and management of big data technologies (e.g., Spark, Hadoop ecosystem).
• Ensure data quality, integrity, and consistency across all data platforms.
• Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver solutions.
• Lead and mentor a team of data engineers, providing technical guidance and fostering best practices.
• Implement data governance policies and ensure compliance with data privacy regulations.
• Automate data processes and monitor system performance, identifying and resolving bottlenecks.
• Evaluate and adopt new data technologies and tools to enhance data infrastructure.
• Contribute to the architectural roadmap for the company's data platform.
• Develop and maintain comprehensive documentation for data pipelines and infrastructure.

Requirements

• Master's degree in Computer Science, Engineering, or a related quantitative field, or equivalent extensive experience.
• 7+ years of experience in data engineering, with at least 3 years in a lead or senior capacity.
• Strong proficiency in SQL and experience with various database technologies (relational and NoSQL).
• Hands-on experience with big data processing frameworks such as Apache Spark, Flink, or Hadoop.
• Expertise in building and managing ETL/ELT pipelines using tools like Airflow, Kafka, or similar.
• Experience with cloud data platforms (e.g., AWS Redshift/S3/EMR, Azure Data Lake/Databricks, GCP BigQuery/Dataflow).
• Proficiency in at least one programming language such as Python, Scala, or Java.
• Understanding of data warehousing concepts, data modeling, and dimensional modeling.
• Experience with data governance and data quality frameworks.
• Strong leadership, communication, and team management skills.
• Ability to work effectively in a hybrid model based in Polokwane .
• Familiarity with containerization technologies (Docker, Kubernetes) is a plus.

Benefits

• A leading annual salary package of up to ZAR.
• Comprehensive medical and life insurance.
• Significant retirement fund contribution.
• Generous performance-based bonuses.
• Opportunities for leadership development and advanced training.
• Flexible working arrangements with a hybrid model in Polokwane .
• Access to cutting-edge data technologies.
• A dynamic and forward-thinking work environment.