Lead Data Engineer - Stellenbosch

Placements24; Stellenbosch; Hybrid; ZAR 1,200,000 - 2,100,000; Full time

Benefits offered by this job

Competitive compensation

Hybrid work environment

Job summary

Placements24 is seeking an experienced Lead Data Engineer to join our Stellenbosch team on a hybrid model. You will oversee the design, development, and optimization of scalable data pipelines and data warehousing solutions, collaborating with data scientists and analysts to deliver robust datasets for advanced analytics and machine learning.

The ideal candidate brings 7+ years in data engineering, with 2+ years in leadership, strong SQL, and experience with Spark, Kafka, Flink, and cloud

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related quantitative field.
• Minimum of 7 years of experience in data engineering, with at least 2 years in a leadership or senior role.
• Proven experience designing and building scalable data pipelines using tools like Spark, Kafka, or Flink.
• Strong expertise in SQL and experience with various database technologies (e.g., PostgreSQL, Snowflake, Redshift).
• Proficiency in cloud data platforms (e.g., AWS Glue, Azure Data Factory, Google Cloud Dataflow).

Responsibilities

• Lead the design, development, and optimization of scalable data pipelines and ETL processes.
• Build and maintain robust data warehousing solutions and data lakes on cloud platforms (e.g., AWS, Azure, GCP).
• Mentor and guide a team of data engineers, fostering best practices and technical growth.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data sets.
• Implement data governance, quality, and security best practices throughout the data lifecycle.
• Develop and maintain data models and schemas for efficient data storage and retrieval.
• Monitor and troubleshoot data pipeline performance, ensuring reliability and efficiency.
• Evaluate and recommend new data technologies and tools to enhance our data capabilities.
• Automate data processes and implement CI/CD for data infrastructure.
• Ensure data solutions meet the needs of the business and support strategic objectives.

Skills

Leadership

Communication

Analytical thinking

Agile experience

Education

Bachelors/Masters in CS/Data Eng

Tools

Spark

Kafka

Flink

PostgreSQL

Snowflake

Redshift

Python

Scala

Job description

About the Role

Our client, a pioneering data analytics firm, is looking for an experienced Lead Data Engineer to join their innovative team in Stellenbosch, Western Cape. This role operates on a hybrid model, combining collaborative office work with the flexibility of remote days. You will lead the design, development, and maintenance of scalable data pipelines and infrastructure that support our client's advanced analytics and machine learning initiatives. As a Lead Data Engineer, you will guide a team of talented data engineers, set technical direction, and ensure the quality and efficiency of data solutions. This position requires a deep understanding of data warehousing, ETL processes, big data technologies, and cloud data platforms. You will work closely with data scientists, analysts, and business stakeholders to understand data needs and deliver robust data solutions. The ideal candidate is a strong communicator, a problem-solver, and passionate about leveraging data to drive business insights. This is an exceptional opportunity to make a significant impact in the rapidly evolving field of data engineering.

Key Responsibilities

• Lead the design, development, and optimization of scalable data pipelines and ETL processes.
• Build and maintain robust data warehousing solutions and data lakes on cloud platforms (e.g., AWS, Azure, GCP).
• Mentor and guide a team of data engineers, fostering best practices and technical growth.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data sets.
• Implement data governance, quality, and security best practices throughout the data lifecycle.
• Develop and maintain data models and schemas for efficient data storage and retrieval.
• Monitor and troubleshoot data pipeline performance, ensuring reliability and efficiency.
• Evaluate and recommend new data technologies and tools to enhance our data capabilities.
• Automate data processes and implement CI/CD for data infrastructure.
• Ensure data solutions meet the needs of the business and support strategic objectives.

Requirements

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related quantitative field.
• Minimum of 7 years of experience in data engineering, with at least 2 years in a leadership or senior role.
• Proven experience designing and building scalable data pipelines using tools like Spark, Kafka, or Flink.
• Strong expertise in SQL and experience with various database technologies (e.g., PostgreSQL, Snowflake, Redshift).
• Proficiency in cloud data platforms (e.g., AWS Glue, Azure Data Factory, Google Cloud Dataflow).
• Experience with programming languages commonly used in data engineering, such as Python or Scala.
• Familiarity with data warehousing concepts and best practices.
• Excellent problem-solving and analytical skills.
• Strong leadership, communication, and interpersonal skills.
• Experience working in an Agile environment.
• Understanding of data architecture principles relevant to Stellenbosch, Western Cape, South Africa.

Benefits

Our client offers a competitive salary, performance bonuses, comprehensive medical and retirement benefits, opportunities for professional development and attending industry conferences, and a dynamic, collaborative hybrid work environment. You will have the opportunity to work on cutting-edge data projects, utilize advanced technologies, and contribute to a company that is at the forefront of data innovation. We foster a culture of learning and growth, providing ample opportunities for career advancement within the field of data engineering.