Data Engineer

Techrepo.co.za; Johannesburg; On-site; ZAR 700,000 - 1,000,000; Full time

Job summary

Nedbank is seeking a data engineer to maintain and build scalable data pipelines and data infrastructure across on-prem and cloud environments. You will work with DB2, PostgreSQL, MSSQL, HBase, NoSQL, and cloud solutions like Azure Databricks, ADF, and ADL Gen 2 to provide secure, governed data for analytics.

You will collaborate with data analysts, software engineers, and data scientists to deliver end-to-end data solutions, optimise performance, and enable data-driven decision making while

Qualifications

• Matric/Grade 12/National Senior Certificate is required.
• Experience designing, building, and maintaining data warehouses/lakes and data pipelines.
• Proficient in Python, Java, SQL with experience in cloud platforms (Azure/AWS/GCP).
• Knowledge of big data technologies (Hadoop, Spark, Hive) and API/ETL tooling.

Responsibilities

• Maintain data pipelines (ingestion, provisioning, streaming, API).
• Build scalable data infrastructure and data models.
• Collaborate with data analysts, software engineers, and data science teams.
• Ensure data quality, security, governance, and compliant data access.
• Drive performance optimization of data warehouses and pipelines.
• Develop APIs to enable data-driven decisions.

Skills

Python

Java

SQL

Big Data

Cloud platforms

Azure

AWS

GCP

NoSQL

ETL tools

Agile Delivery

Problem solving

Communication

Innovation

Data Warehousing

Education

BSc or equivalent in IT/Computing

Cloud certification (Azure/AWS)

Tools

Ab Initio

ADB (Azure Data Bricks)

ADF

SAS ETL

PostgreSQL

MS SQL

IBM DB2

HBase

MongoDB

Azure Data Factory

Azure Databricks

HDInsight

Job description

• Responsible for the maintenance, improvement, cleaning, and manipulation of data in the bank's operational and analytics databases.
• Data Infrastructure: Build and manage scalable, optimised, supported, tested, secure, and reliable data infrastucture eg using Infrastructure and Databases (DB2, PostgreSQL, MSSQL, HBase, NoSQL, etc), Data Lakes Storage (Azure Data Lake Gen 2), Cloud-based solutions (SAS , Azure Databricks, Azure Data Factory, HDInsight), Data Platforms (SAS, Ab Initio, Denodo, Netezza, Azure Cloud). Ensure data security and privacy in collaboration with Information Security, CISO and Data Governance
• Data Pipeline Build (Ingestion, Provisioning, Streaming and API): Build and maintain data pipelines to:
• create data pipelines for data integration (Data Ingestion, Data Provisioning and Data Streaming) utilising both On Premise tool sets and Cloud Data Engineering tool sets
• efficiently extract data (Data Acquisition) from Golden Sources, Trusted sources and Writebacks with data integration from multiple sources, formats and structures
• load the Nedbank Data Warehouse (Data Reservoir, Atomic Data Warehouse, Enterprise Data Mart)
• provide data to the respective Lines of Business Marts, Regulatory Marts and Compliance Marts through self service data virtualisation
• provide data to applications or Nedbank Data consumers
• transform data to a common data model for reporting and data analysis, and to provide data in a consistent, useable format to Nedbank data stakeholders
• handle big data technologies (Hadoop), streaming (KAFKA) and data Replication (IBM Inphosphere Data Replication)
• drive utilisation of data integration tools ( Ab Initio) and Cloud data integration tools (Azure Data Factory and Azure Data Bricks)
• Data Modelling and Schema Build: In collaboration with Data Modellers, create data models and database schemas on the Data Reservoir, Data Lake, Atomic Data Warehouse and Enterprise Data Marts.
• Nedbank Data Warehouse Automation: Automate, monitor and improve the performance of data pipelines.
• Collaboration: Collaborate with Data Analysts, Software Engineers, Data Modelers, Data Scientistsm Scrum Masers and Data Warehouse teams as part of a squad to contribute to the data architecture detail designs and take ownership of Epics end-to-end and ensure that data solutions deliver business value.
• Data Quality and Data Governance: Ensure that reasonable data quality checks are implemented in the data pipelines to maintain a high level of data accuracy, consistency and security.
• Performance and Optimisation: Ensure the performance of the Nedbank data warehouse, integration patterns, batch and real time jobs, streaming and API's.
• API Development: Build API's that enable the Data Driven Organisation, ensuring that the data warehouse is optimised for API's by collaborating with Software Engineers.

Job Description

About the Role

Data Engineer - 146610

Responsibilities

• Information Technology
• Data
• Manager of Self Professional
• Responsible for the maintenance, improvement, cleaning, and manipulation of data in the bank's operational and analytics databases.
• Data Infrastructure: Build and manage scalable, optimised, supported, tested, secure, and reliable data infrastucture eg using Infrastructure and Databases (DB2, PostgreSQL, MSSQL, HBase, NoSQL, etc), Data Lakes Storage (Azure Data Lake Gen 2), Cloud-based solutions (SAS , Azure Databricks, Azure Data Factory, HDInsight), Data Platforms (SAS, Ab Initio, Denodo, Netezza, Azure Cloud). Ensure data security and privacy in collaboration with Information Security, CISO and Data Governance
• Data Pipeline Build (Ingestion, Provisioning, Streaming and API): Build and maintain data pipelines to:
• create data pipelines for data integration (Data Ingestion, Data Provisioning and Data Streaming) utilising both On Premise tool sets and Cloud Data Engineering tool sets
• efficiently extract data (Data Acquisition) from Golden Sources, Trusted sources and Writebacks with data integration from multiple sources, formats and structures
• load the Nedbank Data Warehouse (Data Reservoir, Atomic Data Warehouse, Enterprise Data Mart)
• provide data to the respective Lines of Business Marts, Regulatory Marts and Compliance Marts through self service data virtualisation
• provide data to applications or Nedbank Data consumers
• transform data to a common data model for reporting and data analysis, and to provide data in a consistent, useable format to Nedbank data stakeholders
• handle big data technologies (Hadoop), streaming (KAFKA) and data Replication (IBM Inphosphere Data Replication)
• drive utilisation of data integration tools ( Ab Initio) and Cloud data integration tools (Azure Data Factory and Azure Data Bricks)
• Data Modelling and Schema Build: In collaboration with Data Modellers, create data models and database schemas on the Data Reservoir, Data Lake, Atomic Data Warehouse and Enterprise Data Marts.
• Nedbank Data Warehouse Automation: Automate, monitor and improve the performance of data pipelines.
• Collaboration: Collaborate with Data Analysts, Software Engineers, Data Modelers, Data Scientistsm Scrum Masers and Data Warehouse teams as part of a squad to contribute to the data architecture detail designs and take ownership of Epics end-to-end and ensure that data solutions deliver business value.
• Data Quality and Data Governance: Ensure that reasonable data quality checks are implemented in the data pipelines to maintain a high level of data accuracy, consistency and security.
• Performance and Optimisation: Ensure the performance of the Nedbank data warehouse, integration patterns, batch and real time jobs, streaming and API's.
• API Development: Build API's that enable the Data Driven Organisation, ensuring that the data warehouse is optimised for API's by collaborating with Software Engineers.

Requirements

• Matric / Grade 12 / National Senior Certificate
• Advanced Diplomas/National 1st Degrees
• Total number of years of experience:3 - 6 years
• Experienced at working independently within a squad and has the demonstrated knowledge and skills to deliver data outcomes without supervision.
• Experience designing, building, and maintaining data warehouses and data lakes.
• Experience with big data technologies such as Hadoop, Spark, and Hive.
• Experience with programming languages such as Python, Java, and SQL.
• Experience with relational databases and NoSQL databases.
• Experience with cloud computing platforms such as AWS, Azure, and GCP. Experience with data visualization tools. Result-driven, analytical creative thinker, with demonstrated ability for innovative problem solving.
• Cloud Data Engineering (Azure , AWS, Google)
• Data Warehousing
• Databases (PostgreSQL, MS SQL, IBM DB2, HBase, MongoDB)
• Programming (Python, Java, SQL)
• Data Analysis and Data Modelling
• Data Pipelines and ETL tools (Ab Initio, ADB, ADF, SAS ETL)
• Agile Delivery
• Problem solving skills
• Decision Making
• Influencing
• Communication
• Innovation
• Building Partnerships
• Technical/Professional Knowledge and Skills
• Continuous Learning

Preferred Qualifications

• Field of Study:Bcom, BSc, BEng
• Cloud (Azure, AWS), DEVOPS or Data engineering certification. Any Data Science certification will be an added advantage, Coursera, Udemy, SAS Data Scientist certification, Microsoft Data Scientist.