Data Engineer (Artificial Intelligence)

Indsafri; Johannesburg; On-site; ZAR 600,000 - 1,000,000; Full time

Job summary

Indsafri seeks a highly skilled Data Engineer focused on AI to design, build, and maintain data pipelines and infrastructure that support AI and ML initiatives. You will work with data scientists and ML engineers to ensure data quality, accessibility, and efficiency for model development and deployment.

You will implement governance, monitoring, and cost-efficient data processing, staying current with AI/ML data technologies and best practices.

Qualifications

• Bachelor's or Master's degree in a related field.
• Proven experience as a Data Engineer.
• Strong SQL proficiency with relational and NoSQL databases.
• Expertise in Python, Java, or Scala.
• Experience with Spark, Hadoop, Kafka, or Flink.
• Hands-on cloud experience (AWS, Azure, GCP).
• Familiarity with data warehousing and ETL/ELT processes.
• Understanding of ML concepts and data requirements for AI models.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Responsibilities

• Design, develop, and optimize scalable data pipelines for AI/ML applications.
• Build and maintain data infrastructure (data warehouses, data lakes, real-time systems).
• Implement data governance for accuracy, consistency, and security.
• Collaborate with data scientists and ML engineers on data needs.
• Develop monitoring and alerting for data pipelines and infrastructure.
• Optimize data processing for performance and cost efficiency.
• Stay updated with trends in data engineering, AI, and ML.
• Troubleshoot data-related issues in production.

Skills

SQL proficiency

Analytical thinking

Problem solving

Communication

Team collaboration

Education

Bachelor's or Master's in CS/Engineering/Math

Master's degree preferred

Tools

Python

Java

Scala

Spark

Hadoop

Kafka

Flink

AWS

Azure

GCP

ETL/ELT

Data warehousing

Docker

Kubernetes

Job description

We are seeking a highly skilled and motivated Data Engineer with a focus on Artificial Intelligence (AI) to join our dynamic team. The ideal candidate will be responsible for designing, building, and maintaining robust data pipelines and infrastructure that support our AI and machine learning initiatives. You will work closely with data scientists, ML engineers, and other stakeholders to ensure data quality, accessibility, and efficiency for AI model development and deployment.

Key Responsibilities:

• Design, develop, and optimize scalable data pipelines for ingesting, transforming, and storing large volumes of data for AI/ML applications.
• Build and maintain data infrastructure, including data warehouses, data lakes, and real-time data processing systems.
• Implement data governance best practices to ensure data accuracy, consistency, and security.
• Collaborate with data scientists and ML engineers to understand their data requirements and provide them with clean, well-structured datasets.
• Develop and implement monitoring and alerting systems for data pipelines and infrastructure.
• Optimize data processing for performance and cost-efficiency.
• Stay up-to-date with the latest trends and technologies in data engineering, AI, and machine learning.
• Troubleshoot and resolve data-related issues in production environments.

Required Skills and Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field.
• Proven experience as a Data Engineer or in a similar role.
• Strong proficiency in SQL and experience with relational and NoSQL databases.
• Expertise in at least one major programming language such as Python, Java, or Scala.
• Experience with big data technologies like Spark, Hadoop, Kafka, or Flink.
• Hands-on experience with cloud platforms (AWS, Azure, GCP) and their data services.
• Familiarity with data warehousing concepts and ETL/ELT processes.
• Understanding of machine learning concepts and the data requirements for AI models.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Preferred Qualifications:

• Experience with MLOps principles and tools.
• Familiarity with containerization technologies like Docker and Kubernetes.
• Experience with data modeling and schema design.
• Knowledge of machine learning frameworks (e.g., TensorFlow, PyTorch, scikit-learn).