Senior Data Engineer - Big Data & AI Infrastructure

Placements24; Johannesburg; Hybrid; ZAR 900,000 - 1,600,000; Full time

Benefits offered by this job

Health insurance

Retirement plans

Paid time off

Hybrid work schedule

Professional development

Job summary

Placements24 in Randburg, Gauteng, invites a Senior Data Engineer to design, build, and maintain scalable data pipelines and data warehousing supporting AI initiatives.

Work with Spark, Kafka, Flink and cloud platforms (AWS/Azure/GCP), mentor juniors, and collaborate with data scientists to deliver secure, high-quality data infrastructure.

Qualifications

• 5+ years of professional data engineering experience with big data tech.
• Experience designing and implementing ETL/ELT pipelines (Spark, Kafka, Flink).
• Experience with cloud data platforms (AWS, Azure, GCP).
• Proficient in SQL; familiarity with NoSQL databases.
• Strong Python or Java/Scala programming skills.
• Knowledge of data warehousing concepts and data modeling.

Responsibilities

• Design, build, and maintain scalable data pipelines for large data volumes.
• Develop data warehousing and data lake solutions to support AI/ML workloads.
• Optimize data infrastructure for performance and cost-efficiency.
• Implement data governance, security, and quality standards.
• Collaborate with data scientists and AI engineers to meet data needs.
• Monitor data systems and troubleshoot to ensure high availability.
• Evaluate and integrate new data technologies to enhance the platform.
• Document data pipelines and infrastructure.
• Mentor junior data engineers and contribute to team growth.

Skills

ETL/ELT pipelines

Apache Spark

Kafka

Flink

Python

SQL

NoSQL databases

Cloud platforms

Data warehousing

Data modeling

Docker

Kubernetes

Education

Bachelor's/Master's in CS/Engineering

Tools

AWS Redshift

Azure Synapse

GCP BigQuery

AWS Glue

Apache Spark

Kafka

Job description

About the Role

Our client, a dynamic innovator in AI and emerging technologies, is seeking a highly experienced Senior Data Engineer to build and maintain robust big data infrastructure that powers our advanced AI initiatives. This role is integral to ensuring the seamless flow, storage, and accessibility of massive datasets required for machine learning and analytics. You will be responsible for designing, developing, and optimizing data pipelines and data warehousing solutions. This position is based in Randburg, Gauteng, ZA and offers a flexible hybrid work model, combining the benefits of in-office collaboration with remote flexibility.

Key Responsibilities

• Design, build, and maintain scalable and reliable data pipelines for ingesting, processing, and transforming large volumes of data from various sources.
• Develop and manage data warehousing solutions and data lakes that support AI/ML workloads and business intelligence.
• Optimize data infrastructure for performance, cost-efficiency, and reliability.
• Implement data governance, security, and quality standards.
• Collaborate with data scientists and AI engineers to understand their data needs and provide efficient data solutions.
• Monitor data systems, troubleshoot issues, and implement solutions to ensure high availability.
• Evaluate and integrate new data technologies and tools to enhance our data platform.
• Develop and maintain comprehensive documentation for data pipelines and infrastructure.
• Mentor junior data engineers and contribute to the team's technical growth.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
• 5+ years of professional experience in data engineering, with a strong focus on big data technologies.
• Proven expertise in designing and implementing ETL/ELT pipelines using tools like Apache Spark, Kafka, Flink, or similar.
• Extensive experience with cloud data platforms (e.g., AWS S3, Redshift, Glue; Azure Data Lake, Synapse; GCP BigQuery, Dataflow).
• Proficiency in SQL and experience with NoSQL databases.
• Strong programming skills in Python or Java/Scala.
• Familiarity with data warehousing concepts and data modeling techniques.
• Experience with containerization technologies (Docker, Kubernetes) is a plus.
• Excellent problem-solving skills and the ability to work effectively in a hybrid team environment.
• Strong communication and collaboration abilities.

Benefits

Our client offers a competitive salary, attractive bonuses, and comprehensive benefits including health insurance, retirement plans, and paid time off. Employees benefit from a flexible hybrid work schedule, allowing for a good work-life balance. Access to continuous learning and professional development resources, including training and conferences, is provided. Join a collaborative and innovative environment focused on leveraging data to drive AI advancements.