Lead Data Engineer: Big Data Platform Architect

Placements24; Soweto; On-site; ZAR 1,200,000 - 1,800,000; Full time

Benefits offered by this job

Competitive salary & bonuses

Equity potential

Medical aid

Retirement fund

Paid time off

Professional development

Job summary

Placements24 in Soweto, Gauteng seeks a Principal Data Engineer to design, build, and maintain scalable big data infrastructure supporting analytics and ML initiatives.

You will lead the development of high-performance data pipelines, data warehouses, and data lakes, utilizing Spark, Hadoop, and Flink, while ensuring data quality, security, and governance. Collaboration with data scientists and engineers is essential, with mentorship of juniors and technology evaluation included.

Qualifications

• Advanced degree in a quantitative field with 8+ years in data engineering.
• Strong background in big data platforms and distributed systems.
• Expertise with Spark, Hadoop ecosystem, and data warehousing concepts.
• Proficient in SQL and multiple DB technologies; cloud experience required.

Responsibilities

• Design, develop, and optimize high-performance data pipelines and ETL/ELT operations.
• Build scalable data warehouses and data lakes with modern big data tech.
• Implement distributed processing with Spark, Hadoop, and Flink.
• Develop data models and schemas for analytics and reporting.
• Ensure data quality, security, and governance across platforms.
• Lead and mentor junior data engineers and guide architectural decisions.
• Automate processes and implement robust monitoring and alerts.
• Collaborate with data scientists and developers to meet data needs.
• Evaluate new technologies to enhance data infrastructure.

Skills

Distributed computing

Spark

Hadoop

SQL

Cloud platforms

Python/Scala/Java

Data warehousing

ETL/ELT

Airflow/Luigi

Leadership

Education

MSc or PhD in CS/Engineering/IS

Tools

AWS (EMR, S3, Redshift)

Azure (Databricks, ADLS)

GCP (BigQuery, Dataflow)

Databricks

HDFS

Job description

Placements24 in Soweto, Gauteng seeks a Principal Data Engineer to design, build, and maintain scalable big data infrastructure supporting analytics and ML initiatives.

You will lead the development of high-performance data pipelines, data warehouses, and data lakes, utilizing Spark, Hadoop, and Flink, while ensuring data quality, security, and governance. Collaboration with data scientists and engineers is essential, with mentorship of juniors and technology evaluation included.