Data Engineer - Big Data Platforms

Placements24; Stellenbosch Local Municipality; Hybrid; ZAR 480,000 - 800,000; Full time

Benefits offered by this job

Medical aid

Provident fund

Paid vacation time

Training & conferences

Hybrid work

Job summary

Placements24 seeks an experienced Data Engineer to design, develop, and maintain scalable big data platforms in the Paarl area. You will ingest, transform, and store massive data volumes to enable analytics and ML initiatives.

The role requires strong data architecture, distributed systems, and cloud experience with Python/Scala/Java. You will collaborate with analysts, scientists, and engineers in a dynamic team.

Qualifications

• Bachelor's degree in CS/engineering/IT or related field.
• Proven experience as a Data Engineer with big data tech.
• Strong SQL skills and experience with relational and NoSQL DBs.
• Hands-on with Spark, Hadoop, Kafka or similar.
• Experience with cloud data services (AWS/Azure/GCP).
• Programming in Python, Scala or Java.
• Knowledge of data modeling and ETL/ELT concepts.
• Airflow experience is a plus.
• Excellent problem-solving and collaboration abilities.
• Strong communication skills.

Responsibilities

• Design, build, and maintain scalable big data pipelines.
• Develop and optimize data ingestion, transformation, and loading.
• Implement data warehousing and manage storage systems.
• Ensure data quality, integrity, and security across platforms.
• Collaborate with data scientists and engineers to meet data needs.
• Monitor and troubleshoot pipeline performance and reliability.
• Implement and manage ETL/ELT processes.
• Stay current with emerging big data technologies and best practices.
• Document data architectures, processes, and standards.

Skills

SQL

Python

Scala

Java

Data modeling

ETL/ELT

Big data

Distributed systems

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

Apache Spark

Hadoop

Kafka

Airflow

AWS

Azure

GCP

Job description

About the Role

Our client, a rapidly growing technology firm specializing in data solutions, is seeking an experienced Data Engineer to join their team. This position is based in Paarl, Western Cape , and involves the design, development, and maintenance of robust, scalable big data platforms. You will play a key role in building and optimizing data pipelines that ingest, transform, and store vast amounts of data, enabling advanced analytics and machine learning initiatives across the organization. The ideal candidate has a strong background in data architecture, distributed systems, and programming, with hands-on experience with cloud-based data technologies. This is an excellent opportunity to work with cutting-edge technologies in a dynamic and collaborative environment, contributing directly to the company's data infrastructure and strategy. You will be instrumental in ensuring data availability and quality for critical business decisions.

Key Responsibilities

• Design, build, and maintain scalable and reliable big data pipelines using distributed computing frameworks.
• Develop and optimize data ingestion, transformation, and loading processes.
• Implement data warehousing solutions and manage data storage systems.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver solutions.
• Monitor and troubleshoot data pipeline performance, identifying and resolving issues proactively.
• Implement and manage ETL/ELT processes.
• Stay current with emerging big data technologies and best practices.
• Document data architectures, processes, and standards.

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Data Engineer or in a similar role, with a focus on big data technologies.
• Strong proficiency in SQL and experience with various database technologies (relational, NoSQL).
• Hands-on experience with big data technologies such as Apache Spark, Hadoop, Kafka, or similar.
• Experience with cloud platforms (AWS, Azure, or GCP) and their data services (e.g., S3, Redshift, Data Factory, BigQuery).
• Proficiency in programming languages like Python, Scala, or Java.
• Understanding of data modeling, data warehousing concepts, and ETL/ELT principles.
• Experience with workflow management tools (e.g., Airflow) is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Benefits

Our client offers a competitive salary package, complemented by a comprehensive benefits scheme. This includes medical aid contributions, a provident fund, and generous paid vacation time. We encourage professional development through training programs, workshops, and opportunities to attend industry conferences. The role is situated in the beautiful Paarl, Western Cape region, offering a stimulating work environment. The hybrid nature of this position allows for flexibility between working from home and collaborating in the office. Additional benefits include performance bonuses and a supportive team culture focused on innovation and continuous improvement.