Principal Data Engineer - Big Data Platforms

Placements24; Soweto; On-site; ZAR 1,200,000 - 1,800,000; Full time

Benefits offered by this job

Competitive salary & bonuses

Equity potential

Medical aid

Retirement fund

Paid time off

Professional development

Job summary

Placements24 in Soweto, Gauteng seeks a Principal Data Engineer to design, build, and maintain scalable big data infrastructure supporting analytics and ML initiatives.

You will lead the development of high-performance data pipelines, data warehouses, and data lakes, utilizing Spark, Hadoop, and Flink, while ensuring data quality, security, and governance. Collaboration with data scientists and engineers is essential, with mentorship of juniors and technology evaluation included.

Qualifications

• Advanced degree in a quantitative field with 8+ years in data engineering.
• Strong background in big data platforms and distributed systems.
• Expertise with Spark, Hadoop ecosystem, and data warehousing concepts.
• Proficient in SQL and multiple DB technologies; cloud experience required.

Responsibilities

• Design, develop, and optimize high-performance data pipelines and ETL/ELT operations.
• Build scalable data warehouses and data lakes with modern big data tech.
• Implement distributed processing with Spark, Hadoop, and Flink.
• Develop data models and schemas for analytics and reporting.
• Ensure data quality, security, and governance across platforms.
• Lead and mentor junior data engineers and guide architectural decisions.
• Automate processes and implement robust monitoring and alerts.
• Collaborate with data scientists and developers to meet data needs.
• Evaluate new technologies to enhance data infrastructure.

Skills

Distributed computing

Spark

Hadoop

SQL

Cloud platforms

Python/Scala/Java

Data warehousing

ETL/ELT

Airflow/Luigi

Leadership

Education

MSc or PhD in CS/Engineering/IS

Tools

AWS (EMR, S3, Redshift)

Azure (Databricks, ADLS)

GCP (BigQuery, Dataflow)

Databricks

HDFS

Job description

About the Role

Our client, a leading data analytics firm, is seeking a highly skilled and experienced Principal Data Engineer to join their elite team in Soweto, Gauteng . This critical role will focus on designing, building, and maintaining robust, scalable big data infrastructure and pipelines that support advanced analytics and machine learning initiatives. You will be instrumental in shaping our client's data strategy, working with cutting-edge technologies to ensure efficient data ingestion, processing, and storage. The ideal candidate will possess a deep understanding of distributed systems, data warehousing concepts, and a proven ability to architect complex data solutions. This is an exceptional opportunity to make a significant impact on data-driven decision-making within a forward-thinking organization in the vibrant community of Soweto .

Key Responsibilities

• Design, develop, and optimize high-performance data pipelines for large-scale data processing and ETL/ELT operations.
• Build and maintain scalable data warehouses and data lakes using modern big data technologies.
• Implement and manage distributed data processing frameworks such as Apache Spark, Hadoop, and Flink.
• Develop data models and schemas that support various analytical and reporting needs.
• Ensure data quality, integrity, and security across all data platforms.
• Collaborate with data scientists, analysts, and application developers to understand data requirements and deliver effective solutions.
• Automate data processes and implement robust monitoring and alerting systems.
• Evaluate and recommend new technologies and tools to enhance the data infrastructure.
• Provide technical leadership and mentorship to junior data engineers.
• Contribute to the overall data strategy and architecture roadmap.

Requirements

• M.Sc. or Ph.D. in Computer Science, Engineering, Information Systems, or a related quantitative field.
• A minimum of 8 years of experience in data engineering, with a strong focus on building and managing big data platforms.
• Expertise in distributed computing frameworks like Apache Spark, Hadoop Ecosystem (HDFS, MapReduce, Hive, etc.).
• Proficiency in SQL and experience with various database technologies (relational, NoSQL, columnar stores).
• Hands-on experience with cloud-based data services on platforms such as AWS (EMR, S3, Redshift), Azure (Databricks, ADLS), or GCP (BigQuery, Dataflow).
• Strong programming skills in languages like Python, Scala, or Java.
• Solid understanding of data warehousing concepts, data modeling, and ETL/ELT best practices.
• Experience with data pipeline orchestration tools (e.g., Airflow, Luigi).
• Excellent problem-solving, analytical, and critical thinking skills.
• Strong communication and collaboration abilities to work effectively within our Soweto -based team.

Benefits

• Our client offers a highly competitive salary package, commensurate with experience, along with performance-based bonuses and potential for equity participation.
• Comprehensive benefits include premium medical aid coverage, a generous retirement fund, and ample paid time off.
• Significant investment is made in employee professional development, offering opportunities for advanced training, certifications, and attending cutting-edge industry conferences.
• You will work with state-of-the-art big data technologies and contribute to high-impact data projects.
• The role is based in Gauteng , providing access to a thriving tech ecosystem.