Data Engineer with AI Experience

Indsafri; Johannesburg; On-site; ZAR 600,000 - 900,000; Full time

Job summary

Indsafri in Johannesburg is seeking a Data Engineer with AI experience to design, build and maintain scalable data pipelines and platforms, and collaborate with AI/ML teams to prepare data for AI solutions.

The ideal candidate has 4+ years of data engineering experience, strong Python and SQL skills, and hands-on knowledge of ETL/ELT, cloud platforms, Spark or Kafka, and APIs.

Qualifications

• 4+ years of experience as a Data Engineer.
• Strong Python and SQL skills.
• Experience with ETL/ELT and data pipelines.
• Experience with cloud platforms (AWS, Azure or GCP).
• Experience with Spark, Kafka or similar big-data technologies.
• Understanding of AI/ML data requirements and Generative AI.
• Experience with APIs, vector databases or RAG pipelines is advantageous.
• Strong problem-solving and analytical skills.

Responsibilities

• Design, build and maintain scalable data pipelines and data platforms.
• Develop ETL/ELT processes for structured and unstructured data.
• Work with AI/ML teams to prepare, transform and optimise data for AI solutions.
• Build data pipelines supporting Generative AI, LLMs and AI applications.
• Integrate APIs, databases and cloud data services.
• Implement data quality, governance, security and monitoring.
• Collaborate with Data Scientists, AI Engineers and Software Engineers.

Tools

Python

SQL

ETL/ELT

Data Warehousing

AWS / Azure / GCP

Spark

Kafka

APIs

Vector Databases

RAG pipelines

Databricks

Snowflake

Docker

Kubernetes

MLflow

Job description

Data Engineer with AI Experience

Key Responsibilities:

• Design, build and maintain scalable data pipelines and data platforms.
• Develop ETL/ELT processes for structured and unstructured data.
• Work with AI/ML teams to prepare, transform and optimise data for AI solutions.
• Build data pipelines supporting Generative AI, LLMs and AI applications.
• Integrate APIs, databases and cloud data services.
• Implement data quality, governance, security and monitoring.
• Collaborate with Data Scientists, AI Engineers and Software Engineers.

Requirements:

• 4+ years’ experience as a Data Engineer.
• Strong Python and SQL skills.
• Experience with ETL/ELT, data pipelines and data warehousing.
• Experience with cloud platforms such as AWS, Azure or GCP.
• Experience with Spark, Kafka or similar big-data technologies.
• Understanding of AI/ML data requirements, LLMs or Generative AI.
• Experience working with APIs, vector databases or RAG pipelines is advantageous.
• Strong problem-solving and analytical skills.

Nice to Have:

• Databricks / Snowflake
• Docker & Kubernetes
• MLflow
• Vector databases
• LangChain / LlamaIndex
• Generative AI / LLM integration

Application Notice

Should you not receive any feedback within three (3) weeks of submitting your application, please consider your application unsuccessful.