AI Data Engineer - Large-Scale Data Pipelines

Placements24; East London; Híbrido; ZAR 700.000 - 1.100.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Remote-first culture

Health benefits

Professional development

Descripción de la vacante

Placements24 is seeking an AI Data Engineer to join a fully remote team. You will design, build, and maintain scalable data pipelines powering AI/ML models and applications. The role requires strong data engineering fundamentals, big data tech, and cloud platform experience.

You will collaborate with data scientists, ML engineers, and software engineers to ensure data quality and availability. Remote-first work environment offers flexibility and opportunities to stay at the forefront of AI data

Formación

• Bachelor's degree in a related quantitative field.
• Minimum of 5 years of experience in data engineering focusing on large-scale pipelines.
• Proven experience with big data technologies (Spark, Kafka, Hadoop Ecosystem).
• Strong SQL and programming skills in Python or Scala.
• Experience with cloud data platforms (AWS, Azure, or GCP) and data services.
• Solid understanding of data warehousing, data modeling, and database technologies.
• Familiarity with MLOps is a plus.
• Experience with Docker and Kubernetes is beneficial.

Responsabilidades

• Design, build, and maintain scalable data pipelines for AI/ML workloads.
• Develop ETL/ELT processes to ingest, transform, and load data into data lakes and warehouses.
• Optimize data structures and data flow for performance and cost efficiency.
• Collaborate with data scientists, ML engineers, and software engineers to meet data requirements.
• Implement data quality checks and monitoring to ensure reliability.
• Work with distributed computing frameworks such as Spark, Hadoop.
• Utilize cloud data services (S3, ADLS, GCS, Redshift, BigQuery).
• Implement data governance and security best practices.
• Troubleshoot data pipelines and infrastructure issues.
• Stay updated with data engineering trends.

Conocimientos

Data engineering

SQL

Python

Scala

Spark

Kafka

Hadoop

Cloud platforms

Data warehousing

NoSQL

Educación

Bachelor's degree

Herramientas

Docker

Kubernetes

Descripción del empleo

About the Role

Our client, a cutting-edge technology firm specializing in AI-driven solutions, is looking for a skilled AI Data Engineer to join their fully remote team. This role is critical for designing, building, and maintaining robust, scalable data pipelines that power our advanced machine learning models and AI applications. You will work with vast datasets, ensuring data quality, availability, and accessibility for our data science and engineering teams. The ideal candidate will have a strong background in data engineering principles, big data technologies, and a solid understanding of the AI/ML lifecycle. This is an exceptional opportunity to contribute to impactful projects in a collaborative, remote-first environment.

Key Responsibilities

• Design, construct, install, test, and maintain highly scalable data management systems and data pipelines for AI/ML workloads.
• Develop ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize data structures and data flow for performance, efficiency, and cost-effectiveness.
• Collaborate with data scientists, ML engineers, and software engineers to understand data requirements and deliver data solutions.
• Implement data quality checks and monitoring mechanisms to ensure data accuracy and reliability.
• Work with distributed computing frameworks such as Apache Spark, Hadoop, and related technologies.
• Utilize cloud platforms (AWS, Azure, GCP) and their data services (e.g., S3, ADLS, GCS, Redshift, BigQuery).
• Implement data governance and security best practices.
• Troubleshoot and resolve issues related to data pipelines and infrastructure.
• Stay current with emerging trends and technologies in data engineering and big data.

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
• Minimum of 5 years of experience in data engineering, with a focus on building and managing large-scale data pipelines.
• Proven experience with big data technologies (e.g., Spark, Kafka, Hadoop Ecosystem).
• Strong proficiency in SQL and programming languages such as Python or Scala.
• Experience with cloud data platforms (AWS, Azure, or GCP) and their associated data services.
• Solid understanding of data warehousing concepts, data modeling, and database technologies (relational and NoSQL).
• Familiarity with MLOps principles and tools is a plus.
• Experience with containerization (Docker) and orchestration (Kubernetes) is beneficial.
• Excellent problem-solving, analytical, and debugging skills.
• Strong communication and collaboration skills, especially in a remote setting.
• Ability to work independently and manage multiple priorities effectively.

Benefits

Our client offers a competitive salary, comprehensive health and wellness benefits, and a generous retirement savings plan. As a remote-first company, employees enjoy the flexibility to work from their preferred location. Professional development is supported through ongoing training, access to online learning platforms, and opportunities to attend virtual conferences. The company culture in East London emphasizes innovation, collaboration, and employee well-being, fostering a productive and engaging work environment.