Senior Data Engineer - AI Platforms

Placements24; Riverton (UT); Hybrid; USD 54,000 - 91,000; Full time

Benefits offered by this job

Hybrid work model

Medical aid

Retirement benefits

Training & conferences

Job summary

Placements24 is seeking a Senior Data Engineer for a hybrid role based in Upington, Northern Cape. You will design and maintain scalable data pipelines and cloud infrastructure to power AI/ML initiatives, collaborating with data scientists and engineers to ensure data availability, quality, and security.

The ideal candidate has 5+ years in data engineering, strong SQL, and hands-on experience with AWS/Azure/GCP, Spark, Hadoop, and Kafka.

Qualifications

• Bachelor's or Master's degree in computer science, IT, engineering, or related field.
• 5+ years of data engineering experience with building data pipelines and infrastructure.
• Proven cloud experience with AWS/Azure/GCP and strong SQL skills.
• Experience with Spark, Hadoop, and Kafka.
• Excellent problem solving and collaboration abilities.

Responsibilities

• Design, build, and maintain scalable data pipelines for ingestion, transformation, and loading of large datasets.
• Develop and manage data infrastructure on cloud platforms to support AI/ML workloads.
• Implement data warehousing and optimize data models for performance and accessibility.
• Collaborate with data scientists and AI engineers to meet data requirements.
• Ensure data quality, governance, and security across platforms.
• Automate data processes and implement monitoring and alerting for pipelines.
• Troubleshoot data issues to minimize disruption to AI/ML ops.
• Stay updated on emerging data engineering technologies and best practices.
• Contribute to the overall data strategy and architecture.

Skills

Data Pipelines

Cloud Platforms

SQL

Python

Big Data

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

AWS

Azure

GCP

Spark

Hadoop

Kafka

Docker

Redshift

Job description

About the Role

Our client, a prominent player in the AI & Emerging Technologies sector, is seeking a Senior Data Engineer to join their dynamic team. This role, based in Upington, Northern Cape , is pivotal in building and maintaining robust data pipelines and infrastructure that power our advanced AI and machine learning initiatives. You will be instrumental in ensuring the availability, quality, and scalability of data used by our data scientists and AI engineers. The ideal candidate possesses strong expertise in data architecture, ETL processes, big data technologies, and cloud platforms. Our client is committed to fostering a culture of innovation and collaboration, providing a stimulating work environment where you can make a significant impact. This position offers a hybrid work model, combining the benefits of in-office collaboration with remote flexibility. Join us to contribute to the foundation of cutting-edge AI solutions.

Key Responsibilities

• Design, build, and maintain scalable and reliable data pipelines for ingestion, transformation, and loading (ETL) of large datasets.
• Develop and manage data infrastructure on cloud platforms (e.g., AWS, Azure, GCP) to support AI/ML workloads.
• Implement data warehousing solutions and optimize data models for performance and accessibility.
• Collaborate with data scientists and AI engineers to understand their data requirements and provide robust data solutions.
• Ensure data quality, integrity, and security across all data platforms.
• Automate data processes and implement monitoring and alerting systems for data pipelines.
• Optimize data retrieval and processing for complex analytical queries and machine learning model training.
• Stay current with emerging data engineering technologies and best practices, evaluating their applicability to our platform.
• Troubleshoot and resolve data-related issues, ensuring minimal disruption to AI/ML operations.
• Contribute to the overall data strategy and architecture of the organization.

Requirements

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related quantitative field.
• Minimum of 5 years of experience in data engineering, with a focus on building data pipelines and infrastructure.
• Proven experience with cloud data services (e.g., AWS S3, Redshift, Glue; Azure Data Factory, Synapse; GCP BigQuery, Dataflow).
• Strong proficiency in SQL and experience with big data technologies (e.g., Spark, Hadoop, Kafka).
• Experience with programming languages such as Python for data manipulation and scripting.
• Familiarity with data warehousing concepts and dimensional modeling.
• Experience in designing and implementing ETL/ELT processes.
• Understanding of data governance, data quality, and data security principles.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.
• Experience with containerization (Docker) is a plus.

Benefits

Our client offers a competitive salary, comprehensive medical aid and retirement benefits, and performance incentives. We provide opportunities for continuous learning and professional development, including access to training programs and industry conferences. This role offers a hybrid work arrangement, providing flexibility and fostering collaboration within our Upington, Northern Cape office. Employees benefit from a supportive team culture, access to modern data technologies, and opportunities for career advancement within the exciting field of AI. Join our team and build the data foundations for groundbreakingAI innovation.