Data Engineer

Indsafri; South Africa; Hybrid; ZAR 600,000 - 900,000; Full time

Job summary

Indsafri is seeking a Data Engineer to design, build, and maintain robust data infrastructure. You will develop and optimize data pipelines, including ingestion, streaming, and API solutions, ensuring high-quality data for analytics and AI initiatives.

You will monitor pipelines, ensure security and reliability, and collaborate with Data Analysts to document and validate data assets. A strong focus on SLAs and enterprise architecture will shape daily work.

Qualifications

• Proven experience in data engineering, including data ingestion, transformation, and provisioning (ETL/ELT).
• Strong experience in building and maintaining data pipelines and streaming technologies.
• Solid understanding of data virtualization, SQL, and database management.
• Proficiency in monitoring and managing cloud infrastructure (compute and storage).
• Strong data profiling, quality assurance, and validation skills.
• Ability to provide first-line technical support and troubleshoot complex pipeline issues.
• Excellent collaboration skills with the ability to translate business needs into technical data solutions.

Responsibilities

• Manage day-to-day data pipeline operations, including data profiling, cleaning, configuration, and validation to support data squads and epics.
• Ensure the security, availability, and reliability of the data infrastructure.
• Maintain end-to-end data pipelines, spanning data virtualization, ingestion, and provisioning.
• Ensure monitoring systems are active and pipelines run successfully, with the ability to implement minor configuration changes.
• Develop virtual databases and generate precise data extracts to meet evolving business requirements.
• Partner with Data Analysts to perform data profiling, validation, and comprehensive documentation to support project epics.
• Provide first-line support for the Data Warehouse, ensuring strict adherence to Service Level Agreements (SLAs) regarding data availability and system reliability.
• Monitor and manage cloud-based compute and storage processes to ensure the seamless execution of cloud pipelines.
• Generate daily operational reports to track job success rates and ensure the data warehouse is maintained to enterprise standards.
• Collaborate with business stakeholders to gain domain knowledge, translate business requirements into technical solutions, and optimize business queries.

Skills

ETL/ELT

Data pipelines

Streaming tech

SQL

Cloud infrastructure

Data profiling

Data QA

Troubleshooting

Collaboration

Education

BSc Computer Science/IT

Cloud certifications (AWS/Azure/GCP)

Tools

SQL Server

Airflow

Kafka

Data virtualization tools

Job description

We are seeking the Data Engineer plays a pivotal role in advancing the organization’s technical capabilities by developing and maintaining robust data infrastructure. You will be responsible for building, monitoring, and optimizing data pipelines including ingestion, provisioning, streaming, and API solutions to ensure the availability of clean, high-quality data. By supporting advanced analytics, machine learning, and AI initiatives, you will contribute directly to our strategic goal of becoming a data-driven organization, ensuring all data products align with enterprise architecture standards.

Responsibilities:

• Manage day-to-day data pipeline operations, including data profiling, cleaning, configuration, and validation to support data squads and epics.
• Ensure the security, availability, and reliability of the data infrastructure.
• Maintain end-to-end data pipelines, spanning data virtualization, ingestion, and provisioning.
• Ensure monitoring systems are active and pipelines run successfully, with the ability to implement minor configuration changes.
• Develop virtual databases and generate precise data extracts to meet evolving business requirements.
• Partner with Data Analysts to perform data profiling, validation, and comprehensive documentation to support project epics.
• Provide first-line support for the Data Warehouse, ensuring strict adherence to Service Level Agreements (SLAs) regarding data availability and system reliability.
• Monitor and manage cloud-based compute and storage processes to ensure the seamless execution of cloud pipelines.
• Generate daily operational reports to track job success rates and ensure the data warehouse is maintained to enterprise standards.
• Collaborate with business stakeholders to gain domain knowledge, translate business requirements into technical solutions, and optimize business queries.

Required Skills and Qualifications:

• Proven experience in data engineering, including data ingestion, transformation, and provisioning (ETL/ELT).
• Strong experience in building and maintaining data pipelines and streaming technologies.
• Solid understanding of data virtualization, SQL, and database management.
• Proficiency in monitoring and managing cloud infrastructure (compute and storage).
• Strong data profiling, quality assurance, and validation skills.
• Ability to provide first-line technical support and troubleshoot complex pipeline issues.
• Excellent collaboration skills with the ability to translate business needs into technical data solutions.

Preferred Qualifications:

• Relevant degree in Computer Science, Information Technology, or a related quantitative field.
• Experience working within an Agile/Scrum environment (Data Squads/Epics).
• Exposure to Big Data ecosystems and enterprise data architecture roadmaps.
• Certification in cloud platforms (e.g., AWS, Azure, or GCP).
• Join a forward-thinking, data-centric team committed to driving technical innovation.
• Opportunities to work on large-scale enterprise data initiatives and cutting-edge AI/ML-enabling infrastructure.
• Collaborative, cross-functional environment fostering technical growth and thought leadership.

Skills

ETL SQL validation Database Management Data Pipelines Quality Assurance ELT cloud infrastructure data profiling Data Engineering Streaming Technologies Data Virtualization