Senior Data Engineer Century City

HR On Site; Milnerton; On-site; ZAR 650,000 - 900,000; Full time

Job summary

HR On Site is seeking an experienced Data Engineer to join a cross-functional agile team. The role focuses on building and maintaining data solutions, ETL processes, and reports while ensuring data availability and integrity.

You will design and deploy data pipelines, optimize data flows, and collaborate with business stakeholders to implement data architecture initiatives in a cloud environment.

Qualifications

• Bachelor’s degree in data management or related field is required.
• Minimum 5 years of data engineering experience with visual pipeline designers (SSIS, Talend, Informatica or Azure Data Factory).
• At least 2 years in a commercial cloud environment like Azure.
• Strong SQL proficiency and query optimization skills.
• Experience with multiple DBMS systems (PostgreSQL, MySQL, MSSQL).
• Hands-on ETL development, data loading from diverse sources, and data governance.
• Familiarity with source control systems and master data management.
• Experience with Bash/PowerShell scripting and cloud data technologies.
• Certifications from Google/Azure/AWS are advantageous.
• Interest in healthcare IT space is a plus.

Responsibilities

• Design, develop, schedule, deploy and monitor data pipelines.
• Optimize, improve, and validate data pipelines.
• Design and deploy data tables, forms, reports, and queries.
• Produce ad-hoc queries and develop reports for business needs.
• Develop security-aware cloud data solutions.
• Process data into formats like Parquet, JSON, Avro.
• Apply engines for data processing including Graph, Relational, NoSQL.
• Create and maintain technical documentation.
• Contribute to data architecture initiatives and strategies.
• Engage with business reps on data solution initiatives.

Skills

SQL proficiency

SQL optimization

ETL development

Data governance

Version control

Scripting (Bash/PowerShell)

Data pipelines

Education

Bachelor’s degree in data management

Tools

SSIS

Talend

Informatica

Azure Data Factory

PostgreSQL

MySQL

MSSQL

Azure data technologies

Job description

Role Overview

The role operates within a cross-functional, agile development team working across various technologies. The key responsibilities of this role will focus on creating and maintaining data solutions. This will include development activities such as the implementation of ETL processes, the creation of reports, ensuring high levels of data availability and integrity. We are therefore seeking an energetic and committed individual that is a self-starter and who works well independently as well as within a team context. Your knowledge of, and experience in this area, together with your ability to learn new technologies, will be invaluable and of the utmost importance to help ensure progress and the successful implementation of these solutions.

Duties & Responsibilities

1. Design, develop, schedule, deploy and monitor data pipelines
2. Optimize, improve, and validate data pipelines
3. Design and deploy data table structures, forms, reports, and queries
4. Produce ad-hoc queries and develop reports to support business needs
5. Develop security aware cloud data solutions
6. Process data into various storage formats such as Parquet, JSON, Avro
7. Apply appropriate engines to data processing such as Graph, Relational, NoSQL
8. Create and maintain technical documentation
9. Help inform and drive data architecture initiatives and strategies
10. Engage with business representatives regarding data solution initiatives

Desired Experience & Qualifications

1. Bachelor’s degree with a focus on data management (e.g. Information Systems, Statistics, Data Engineering)
2. A minimum of 5 years' experience working as a data engineer utilizing visual pipeline designers such as SSIS, Talend, Informatica or Azure Data Factory
3. A minimum of 2 years data engineering in a commercial cloud environment like Azure
4. Advanced proficiency in the SQL language
5. SQL Query optimization skills
6. Experience in at least two DBMS's such as Postgres, MySQL or MSSQL
7. Experience in loading data from diverse sources
8. Experience in the development, implementation and monitoring of ETL processes
9. Experience in database administration, maintenance and optimization
10. A sound understanding of data management principles including data quality assurance and data governance
11. Experience in source code and version control systems
12. Experience in master data management and the implementation and maintenance thereof
13. Experience in Azure data technologies is very advantageous
14. Database administration
15. Bash or PowerShell scripting
16. Data Engineering certifications from Google, Azure or AWS
17. Demonstrated interest in the claims and healthcare IT space

Interested?

We encourage all candidates to read through the job description clearly and properly before applying and submitting your CV for perusal. We appreciate all submissions and will reply to each one as soon as possible.