AI Data Engineer: Scalable Big Data Pipelines in the Cloud

Placements24; Polokwane; On-site; ZAR 600,000 - 950,000; Full time

Benefits offered by this job

Health insurance

Paid time off

Professional development

Company events

Job summary

Placements24 in Polokwane, Limpopo, is seeking an AI Data Engineer to design and maintain scalable data pipelines for AI projects. You will work with large datasets, ensure quality, and enable data scientists and ML engineers with reliable access to data on cloud platforms.

You will implement ETL/ELT processes, build data warehouses and data lakes, and apply data governance and security best practices. The role requires strong SQL, Spark/Hadoop/Flink experience, and proficiency in Python, Scala,

Qualifications

• Bachelor's or Master's degree in CS, Engineering, or related field.
• Proven experience as a Data Engineer with big data tech.
• Strong SQL proficiency and experience with relational and NoSQL databases.
• Hands-on with Spark, Hadoop, or Flink.
• Experience with cloud platforms (AWS, Azure, GCP) and data services.
• Programming in Python, Scala, or Java.
• Experience with Airflow or Luigi.
• Solid data warehousing concepts and data modeling.
• Familiarity with AI/ML data needs.
• Strong problem-solving skills.

Responsibilities

• Design, build and optimize scalable data pipelines for AI/ML applications.
• ETL/ELT processes to ingest, transform, and load data.
• Develop and maintain data warehouses and data lakes on cloud platforms.
• Ensure data quality, integrity, and reliability across all data systems.
• Collaborate with AI Data Scientists and ML Engineers to understand data requirements.
• Implement data governance and security best practices.
• Monitor data pipeline performance and troubleshoot issues proactively.
• Optimize data storage and processing for cost-efficiency and performance.
• Automate data-related tasks and processes using scripting and orchestration tools.
• Stay current with emerging big data technologies and best practices in data engineering.

Skills

SQL

Relational DB

NoSQL DB

Spark

Hadoop

Flink

Cloud AWS

Azure

GCP

Python

Scala

Java

Data Warehousing

Data Modeling

AI/ML concepts

Problem Solving

Education

Bachelor's degree

Master's degree

Tools

Airflow

Luigi

Job description

Placements24 in Polokwane, Limpopo, is seeking an AI Data Engineer to design and maintain scalable data pipelines for AI projects. You will work with large datasets, ensure quality, and enable data scientists and ML engineers with reliable access to data on cloud platforms.

You will implement ETL/ELT processes, build data warehouses and data lakes, and apply data governance and security best practices. The role requires strong SQL, Spark/Hadoop/Flink experience, and proficiency in Python, Scala,