AI Data Engineer - Big Data Pipelines

Placements24; Polokwane; On-site; ZAR 600,000 - 950,000; Full time

Benefits offered by this job

Health insurance

Paid time off

Professional development

Company events

Job summary

Placements24 in Polokwane, Limpopo, is seeking an AI Data Engineer to design and maintain scalable data pipelines for AI projects. You will work with large datasets, ensure quality, and enable data scientists and ML engineers with reliable access to data on cloud platforms.

You will implement ETL/ELT processes, build data warehouses and data lakes, and apply data governance and security best practices. The role requires strong SQL, Spark/Hadoop/Flink experience, and proficiency in Python, Scala,

Qualifications

• Bachelor's or Master's degree in CS, Engineering, or related field.
• Proven experience as a Data Engineer with big data tech.
• Strong SQL proficiency and experience with relational and NoSQL databases.
• Hands-on with Spark, Hadoop, or Flink.
• Experience with cloud platforms (AWS, Azure, GCP) and data services.
• Programming in Python, Scala, or Java.
• Experience with Airflow or Luigi.
• Solid data warehousing concepts and data modeling.
• Familiarity with AI/ML data needs.
• Strong problem-solving skills.

Responsibilities

• Design, build and optimize scalable data pipelines for AI/ML applications.
• ETL/ELT processes to ingest, transform, and load data.
• Develop and maintain data warehouses and data lakes on cloud platforms.
• Ensure data quality, integrity, and reliability across all data systems.
• Collaborate with AI Data Scientists and ML Engineers to understand data requirements.
• Implement data governance and security best practices.
• Monitor data pipeline performance and troubleshoot issues proactively.
• Optimize data storage and processing for cost-efficiency and performance.
• Automate data-related tasks and processes using scripting and orchestration tools.
• Stay current with emerging big data technologies and best practices in data engineering.

Skills

SQL

Relational DB

NoSQL DB

Spark

Hadoop

Flink

Cloud AWS

Azure

GCP

Python

Scala

Java

Data Warehousing

Data Modeling

AI/ML concepts

Problem Solving

Education

Bachelor's degree

Master's degree

Tools

Airflow

Luigi

Job description

About the Role

Our client, a prominent leader in data-driven innovation, is seeking a highly skilled AI Data Engineer to join their energetic team in Polokwane, Limpopo . This role is central to building and maintaining robust, scalable data pipelines essential for AI and machine learning initiatives. You will work with vast datasets, ensuring their quality, availability, and accessibility for data scientists and ML engineers. The ideal candidate possesses strong technical expertise in big data technologies, data warehousing, and cloud infrastructure. You will play a critical role in architecting and implementing solutions that support the company's advanced AI projects. Our client fosters a culture of collaboration and continuous learning, providing ample opportunities to work with cutting-edge technologies. This is an excellent opportunity to contribute to impactful projects in a growing field, based in a key regional hub. You will be responsible for the end-to-end data lifecycle, from ingestion to transformation and storage, ensuring optimal performance and reliability.

Key Responsibilities

• Design, build, and optimize scalable data pipelines for AI and machine learning applications.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources.
• Develop and maintain data warehouses and data lakes on cloud platforms.
• Ensure data quality, integrity, and reliability across all data systems.
• Collaborate with AI Data Scientists and ML Engineers to understand data requirements.
• Implement data governance and security best practices.
• Monitor data pipeline performance and troubleshoot issues proactively.
• Optimize data storage and processing for cost-efficiency and performance.
• Automate data-related tasks and processes using scripting and orchestration tools.
• Stay current with emerging big data technologies and best practices in data engineering.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
• Proven experience as a Data Engineer with a focus on big data technologies.
• Strong proficiency in SQL and experience with relational and NoSQL databases.
• Hands-on experience with big data processing frameworks such as Spark, Hadoop, or Flink.
• Experience with cloud platforms (AWS, Azure, GCP) and their data services (e.g., S3, Redshift, BigQuery, Azure Data Lake).
• Proficiency in at least one programming language such as Python, Scala, or Java.
• Experience with data pipeline orchestration tools (e.g., Airflow, Luigi).
• Solid understanding of data warehousing concepts and data modeling techniques.
• Familiarity with AI/ML concepts and the data needs of ML pipelines.
• Strong problem-solving skills and attention to detail.

Benefits

• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Generous paid time off and company holidays.
• Opportunities for professional development and training in cutting-edge data technologies.
• Access to modern infrastructure and tools in Polokwane .
• A dynamic and collaborative team environment.
• Retirement savings plan.
• Contribution to innovative AI projects.
• Potential for career advancement within the company.
• Regular team events and company-sponsored activities.