Senior Data Engineer (Big Data Platforms)

Placements24; Pretoria; Hybrid; ZAR 900.000 - 1.500.000; Vollzeit

Benefits dieser Stelle

Hybrid work model

Professional development opportunities

Conferences and training

Zusammenfassung

WhatJobs Direct is seeking a Senior Data Engineer in Pretoria, Gauteng. You will build scalable data pipelines using Spark/Hadoop, develop ETL/ELT processes, and manage data warehouses and lakes to support analytics.

Role requires strong SQL, data warehousing, and cloud experience with AWS or similar platforms. Collaboration with data scientists and engineers is essential in a hybrid work environment.

Qualifikationen

• Degree in a quantitative field and proven data engineering experience.
• Strong background with big data technologies and data pipelines.
• Proficiency in Python, Scala, or Java for ETL/ELT development.
• Hands-on with Spark, Hadoop ecosystem, and Kafka or Flink.

Aufgaben

• Design, build, and maintain scalable data pipelines using Spark/Hadoop.
• Develop and optimize ETL/ELT processes for ingestion and transformation.
• Build data warehouses and data lakes for reporting and analytics.
• Ensure data quality, integrity, and accessibility across platforms.
• Collaborate with data scientists and analysts to meet data needs.
• Implement data governance policies and monitor performance.

Kenntnisse

Python/Scala/Java

Big data

SQL

Data warehousing

Data governance

Collaboration

Ausbildung

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Spark

Hadoop

Kafka

Hive

AWS EMR

Jobbeschreibung

About the Role

Our client, WhatJobs Direct, is seeking a highly experienced Senior Data Engineer to join their innovative team based in Pretoria, Gauteng. This role is instrumental in building and maintaining robust, scalable data pipelines and infrastructure that support advanced analytics and business intelligence initiatives. You will work with large, complex datasets, leveraging cutting-edge big data technologies to extract, transform, and load data efficiently. The ideal candidate possesses a strong background in data warehousing, ETL processes, and distributed computing systems. You will collaborate closely with data scientists, analysts, and business stakeholders to ensure data availability, quality, and accessibility. This is an excellent opportunity to make a significant impact in a data-driven organization, contributing to strategic decision-making through reliable and insightful data solutions. Our client is committed to fostering a collaborative environment where continuous learning and technical growth are prioritized.

Key Responsibilities

• Design, build, and maintain scalable and reliable data pipelines using big data technologies (e.g., Spark, Hadoop, Kafka).


• Develop and optimize ETL/ELT processes for data ingestion, transformation, and aggregation.


• Build and manage data warehouses and data lakes to support reporting and analytics needs.


• Ensure data quality, integrity, and consistency across all data platforms.


• Collaborate with data scientists and analysts to understand their data requirements and provide solutions.


• Develop and implement data governance policies and procedures.


• Monitor data pipeline performance and troubleshoot issues proactively.


• Optimize data storage and query performance for large datasets.


• Stay current with emerging trends and technologies in data engineering and big data.


• Contribute to architectural decisions related to data infrastructure and strategy within the Pretoria, Gauteng office.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related quantitative field.


• Proven experience as a Data Engineer, with a strong focus on big data technologies.


• Proficiency in programming languages such as Python, Scala, or Java.


• Hands-on experience with distributed data processing frameworks like Apache Spark and Hadoop ecosystem (HDFS, MapReduce, Hive).


• Experience with stream processing technologies such as Kafka or Flink.


• Solid understanding of SQL and database design principles.


• Experience with cloud data platforms (e.g., AWS EMR, S3, Redshift; Azure Data Factory, HDInsight; GCP Dataflow, BigQuery).


• Knowledge of data warehousing concepts and best practices.


• Excellent problem-solving and analytical skills.


• Strong communication and collaboration skills for working in a hybrid environment.

Benefits

• Competitive salary and performance incentives.


• Comprehensive health and retirement benefits.


• Generous annual leave allowance.


• Opportunities for professional development and attending industry conferences.


• Collaborative and supportive team culture.


• Hybrid work model balancing remote flexibility and office collaboration.


• Access to modern technology stack and tools.


• Employee wellness programs.


• Company-sponsored training and certifications.