Senior Data Engineer

HR On Site; Milnerton; On-site; ZAR 600,000 - 900,000; Full time

Job summary

HR On Site is seeking a seasoned Data Engineer to design, build, and monitor data pipelines and data solutions. You will work across cloud data services, optimize SQL queries, and implement ETL processes that ensure data availability and quality for business insights.

The role requires 5+ years in data engineering, experience with Azure and multiple DBMS, and strong scripting and documentation skills. This is an on-site position based in Milnerton, Western Cape.

Qualifications

• Bachelor’s degree in a data-focused field or equivalent.
• Minimum 5 years as a data engineer with visual pipeline designers (SSIS, Talend, Informatica, or Azure Data Factory).
• Minimum 2 years data engineering in a commercial cloud environment like Azure.
• Advanced SQL skills and query optimization experience.
• Experience with multiple DBMS (PostgreSQL, MySQL, MSSQL).
• Experience in ETL development, data loading from diverse sources, and data governance.

Responsibilities

• Design, develop, schedule, deploy, and monitor data pipelines.
• Optimize, improve, and validate data pipelines.
• Design and deploy data table structures, forms, reports, and queries.
• Produce ad-hoc queries and develop reports for business needs.
• Develop security-aware cloud data solutions.
• Process data into Parquet, JSON, Avro formats.
• Apply engines to data processing (Graph, Relational, NoSQL).
• Create and maintain technical documentation.
• Support data architecture initiatives and strategies.
• Engage with business stakeholders on data initiatives.

Skills

SQL proficiency

Query optimization

Cloud data engineering

ETL development

Data governance

Version control

Scripting (PowerShell/Bash)

Education

Bachelor’s degree in data management / information systems / statistics / data engineering

Tools

SSIS

Talend

Informatica

Azure Data Factory

Job description

Role Overview

The role operates within a cross-functional, agile development team working across various technologies. The key responsibilities of this role will focus on creating and maintaining data solutions, including development activities such as the implementation of ETL processes, the creation of reports, and ensuring high levels of data availability and integrity. We are seeking an energetic and committed individual who is a self-starter and works well independently as well as within a team context. Your knowledge of and experience in this area, together with your ability to learn new technologies, will be invaluable to ensure progress and the successful implementation of these solutions.

Duties & Responsibilities

1. Design, develop, schedule, deploy, and monitor data pipelines
2. Optimize, improve, and validate data pipelines
3. Design and deploy data table structures, forms, reports, and queries
4. Produce ad-hoc queries and develop reports to support business needs
5. Develop security-aware cloud data solutions
6. Process data into various storage formats such as Parquet, JSON, Avro
7. Apply appropriate engines to data processing such as Graph, Relational, NoSQL
8. Create and maintain technical documentation
9. Help inform and drive data architecture initiatives and strategies
10. Engage with business representatives regarding data solution initiatives

Desired Experience & Qualifications

1. Bachelor’s degree with a focus on data management (e.g. Information Systems, Statistics, Data Engineering)
2. A minimum of 5 years' experience working as a data engineer utilizing visual pipeline designers such as SSIS, Talend, Informatica, or Azure Data Factory
3. A minimum of 2 years of data engineering in a commercial cloud environment like Azure
4. Advanced proficiency in SQL
5. SQL Query optimization skills
6. Experience in at least two DBMS's such as Postgres, MySQL, or MSSQL
7. Experience in loading data from diverse sources
8. Experience in the development, implementation, and monitoring of ETL processes
9. Experience in database administration, maintenance, and optimization
10. A sound understanding of data management principles including data quality assurance and data governance
11. Experience in source code and version control systems
12. Experience in master data management and its implementation and maintenance
13. Experience in Azure data technologies, database administration, Bash or PowerShell scripting, and data engineering certifications from Google, Azure, or AWS is advantageous
14. Demonstrated interest in the claims and healthcare IT space

Interested?

We encourage all candidates to read through the job description clearly and properly before applying and submitting your CV for perusal. We appreciate all submissions and will reply to each one as soon as possible.