Lead Data Engineer: Cloud Pipelines & Big Data

Dariel; Gauteng; On-site; ZAR 600,000 - 900,000; Full time

Job summary

Dariel is seeking a Data Engineer to design, build and support scalable data pipelines within an agile team. You will handle raw data, ensure security and repeatable data flows, and drive insights from large datasets.

The role involves building big data solutions with PySpark and Hadoop technologies, implementing ETL processes, and translating business requirements into robust data architectures for analytics and BI readiness.

Qualifications

• Bachelors degree or equivalent in a relevant field is required.
• AWS Certification is preferred and strongly valued.
• Strong programming or scripting language knowledge across data workloads.
• Ability to model data and understand varied structures for use cases.

Responsibilities

• Architect data analytics framework and scalable pipelines.
• Translate complex requirements into architecture and code.
• Lead batch and real-time analytical solutions with modern tech.
• Develop data feeds from on-premise to AWS Cloud and maintain production feeds.
• Create data marts using ETL tools; manage Databases and operations.

Skills

Programming languages

Data modelling

Distributed systems

Python

PySpark

Spark

Big Data

Education

Bachelor’s Degree in Computer Science, Computer Engineering, or equivalent

AWS Certification

Tools

Talend

AWS EMR

EC2

S3

Job description

Dariel is seeking a Data Engineer to design, build and support scalable data pipelines within an agile team. You will handle raw data, ensure security and repeatable data flows, and drive insights from large datasets.

The role involves building big data solutions with PySpark and Hadoop technologies, implementing ETL processes, and translating business requirements into robust data architectures for analytics and BI readiness.