Senior Data Engineer (Big Data & ETL)

Placements24; Sandton; Hybrid; ZAR 900,000 - 1,600,000; Full time

Benefits offered by this job

Competitive salary

Hybrid work arrangement

Professional development opportunities

Challenging data projects

Collaborative environment

Job summary

Placements24 client, a dynamic data analytics firm in Sandton, Gauteng, is seeking a Senior Data Engineer to design, build and maintain scalable data pipelines and infrastructure for BI, ML and reporting initiatives.

You will work with ETL/ELT pipelines, big data technologies (Spark, Hadoop), and cloud platforms (AWS, Azure, GCP), collaborating with data scientists and business analysts to deliver reliable data solutions. Hybrid work model offered.

Qualifications

• Minimum 5 years of experience in data engineering or a related field.
• Proven experience designing and implementing complex ETL/ELT pipelines.
• Strong proficiency in SQL and experience with relational databases (PostgreSQL, SQL Server).
• Hands-on experience with Spark and distributed systems.
• Experience with cloud data platforms (AWS Redshift, S3, Glue; Azure Data Factory, Synapse; GCP BigQuery, Dataflow).
• Proficiency in Python or Scala for data manipulation and pipeline development.
• Familiarity with data warehousing concepts and best practices.
• Experience with data modeling and schema design.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Responsibilities

• Design, develop, and optimize robust ETL/ELT data pipelines to ingest, transform, and load data into data warehouses and data lakes.
• Build and maintain scalable data processing systems using big data technologies (e.g., Apache Spark, Hadoop).
• Develop and manage data models and database schemas for optimal performance and usability.
• Implement data quality checks and validation processes to ensure data accuracy and consistency.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions.
• Monitor data pipeline performance, troubleshoot issues, and implement solutions for improved efficiency and reliability.
• Develop and maintain data infrastructure on cloud platforms (e.g., AWS, Azure, GCP).
• Stay current with emerging data engineering technologies and best practices.
• Document data architectures, pipelines, and processes.

Skills

SQL

PostgreSQL

SQL Server

Apache Spark

Hadoop

Python

Scala

Data warehousing

Data modeling

ETL/ELT

Cloud platforms

Problem solving

Communication

Tools

Spark/Hadoop

AWS Redshift/S3/Glue

Azure Data Factory/Synapse

GCP BigQuery/Dataflow

Job description

About the Role

Our client, a dynamic data analytics firm, is seeking a skilled Senior Data Engineer to join their team in **Sandton, Gauteng**. This role offers a hybrid work model, blending office collaboration with remote flexibility. You will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure that enable efficient data processing and analysis. This is a critical role in ensuring the availability and integrity of data for business intelligence, machine learning, and reporting initiatives. The ideal candidate will have a strong background in ETL processes, big data technologies, and cloud data platforms. You'll be working with a team of data professionals committed to leveraging data for strategic decision-making.

Key Responsibilities

• Design, develop, and optimize robust ETL/ELT data pipelines to ingest, transform, and load data from various sources into data warehouses and data lakes.
• Build and maintain scalable data processing systems using big data technologies (e.g., Apache Spark, Hadoop).
• Develop and manage data models and database schemas for optimal performance and usability.
• Implement data quality checks and validation processes to ensure data accuracy and consistency.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions.
• Monitor data pipeline performance, troubleshoot issues, and implement solutions for improved efficiency and reliability.
• Develop and maintain data infrastructure on cloud platforms (e.g., AWS, Azure, GCP).
• Stay current with emerging data engineering technologies and best practices.
• Document data architectures, pipelines, and processes.

Requirements

• Minimum of 5 years of experience in data engineering or a related field.
• Proven experience designing and implementing complex ETL/ELT pipelines.
• Strong proficiency in SQL and experience with relational databases (e.g., PostgreSQL, SQL Server).
• Hands-on experience with big data technologies such as Apache Spark and distributed systems.
• Experience with cloud data platforms (e.g., AWS Redshift, S3, Glue; Azure Data Factory, Synapse; GCP BigQuery, Dataflow).
• Proficiency in programming languages like Python or Scala for data manipulation and pipeline development.
• Familiarity with data warehousing concepts and best practices.
• Experience with data modeling and schema design.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Benefits

• Competitive salary and potential for performance bonuses.
• Hybrid work arrangement for a flexible work-life balance.
• Opportunities for professional development and acquiring new skills.
• Work on challenging and impactful data projects.
• Collaborative environment with experienced data professionals.
• Exposure to modern big data technologies and cloud solutions.
• Comprehensive health and retirement benefits.