Senior Data Engineer - AI Platforms

Placements24; Vrystaat; Hybrid; ZAR 900,000 - 1,500,000; Full time

Benefits offered by this job

Competitive salary

Medical aid and retirement benefits

Training programs and conferences

Hybrid work arrangement

Job summary

Placements24 is seeking a Senior Data Engineer to design and maintain scalable data pipelines and cloud data infrastructure for AI/ML workloads in Upington, Northern Cape. The role focuses on data availability, quality, and security, collaborating with data scientists and engineers in a hybrid work model.

You will implement ETL/ELT processes, develop data warehousing solutions, and optimize data models for performance across AWS, Azure, and GCP environments, using Spark, Hadoop, and SQL.

Qualifications

• Bachelor's or Master's in CS/IT/Engineering or related quantitative field.
• Minimum 5 years' experience in data engineering and pipelines.
• Proven cloud data services experience across AWS/Azure/GCP.
• Strong SQL, and big data tech like Spark, Hadoop, Kafka.
• Python for data manipulation and scripting; ETL/ELT design.
• Understanding of data governance, quality, and security.

Responsibilities

• Design, build, and maintain scalable ETL data pipelines for large datasets.
• Develop and manage cloud-based data infrastructure for AI/ML workloads.
• Implement data warehousing and optimize models for performance.
• Collaborate with data scientists and AI engineers to meet data needs.
• Ensure data quality, integrity, and security across platforms.
• Automate data processes and monitor pipelines with alerts.
• Optimize data retrieval for complex analytics and model training.
• Stay current with emerging data engineering technologies.
• Troubleshoot data-related issues to minimize AI/ML disruption.
• Contribute to overall data strategy and architecture.

Skills

SQL

Python

Spark

Hadoop

Kafka

Docker

Data Warehousing

Cloud Platforms

Education

Bachelor's or Master's degree in CS/IT/Engineering

Master's degree preferred

Tools

AWS

Azure

GCP

Redshift

BigQuery

Glue

Synapse

Data Factory

Dataflow

Docker

Job description

About the Role

Our client, a prominent player in the AI & Emerging Technologies sector, is seeking a Senior Data Engineer to join their dynamic team. This role, based in Upington, Northern Cape , is pivotal in building and maintaining robust data pipelines and infrastructure that power our advanced AI and machine learning initiatives. You will be instrumental in ensuring the availability, quality, and scalability of data used by our data scientists and AI engineers. The ideal candidate possesses strong expertise in data architecture, ETL processes, big data technologies, and cloud platforms. Our client is committed to fostering a culture of innovation and collaboration, providing a stimulating work environment where you can make a significant impact. This position offers a hybrid work model, combining the benefits of in-office collaboration with remote flexibility. Join us to contribute to the foundation of cutting-edge AI solutions.

Key Responsibilities

• Design, build, and maintain scalable and reliable data pipelines for ingestion, transformation, and loading (ETL) of large datasets.
• Develop and manage data infrastructure on cloud platforms (e.g., AWS, Azure, GCP) to support AI/ML workloads.
• Implement data warehousing solutions and optimize data models for performance and accessibility.
• Collaborate with data scientists and AI engineers to understand their data requirements and provide robust data solutions.
• Ensure data quality, integrity, and security across all data platforms.
• Automate data processes and implement monitoring and alerting systems for data pipelines.
• Optimize data retrieval and processing for complex analytical queries and machine learning model training.
• Stay current with emerging data engineering technologies and best practices, evaluating their applicability to our platform.
• Troubleshoot and resolve data-related issues, ensuring minimal disruption to AI/ML operations.
• Contribute to the overall data strategy and architecture of the organization.

Requirements

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related quantitative field.
• Minimum of 5 years of experience in data engineering, with a focus on building data pipelines and infrastructure.
• Proven experience with cloud data services (e.g., AWS S3, Redshift, Glue; Azure Data Factory, Synapse; GCP BigQuery, Dataflow).
• Strong proficiency in SQL and experience with big data technologies (e.g., Spark, Hadoop, Kafka).
• Experience with programming languages such as Python for data manipulation and scripting.
• Familiarity with data warehousing concepts and dimensional modeling.
• Experience in designing and implementing ETL/ELT processes.
• Understanding of data governance, data quality, and data security principles.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.
• Experience with containerization (Docker) is a plus.

Benefits

• Our client offers a competitive salary, comprehensive medical aid and retirement benefits, and performance incentives.
• We provide opportunities for continuous learning and professional development, including access to training programs and industry conferences.
• This role offers a hybrid work arrangement, providing flexibility and fostering collaboration within our Upington, Northern Cape office.