Intermediate Data Engineer

Boardroom Appointments; Johannesburg; On-site; ZAR 700,000 - 1,000,000; Full time

Job summary

Boardroom Appointments seeks an Intermediate Data Engineer in Johannesburg to advance scalable data pipelines and analytics capabilities. The role involves building robust ETL processes and ensuring data quality across the organization.

The ideal candidate has 3+ years experience with SQL/NoSQL, Python, AWS, Databricks, and data warehousing, working with analysts and data scientists to drive transformative insights.

Qualifications

• Postgraduate IT degree required.
• Masters in IT preferred.
• Strong data engineering foundations and programming skills.

Responsibilities

• Identify shortcomings and deliver a plan for small change initiatives with PM guidance.
• Build analytics tools using the data pipeline to provide insights on acquisition and efficiency.
• Monitor metrics, analyze data, and collaborate with Data/Analytics teams to improve systems.
• Automate manual processes and redesign infrastructure for scalability.
• Develop ETL processes to convert data for consumption.
• Consider risk, regulatory, prudential and compliance needs.
• Collaborate with data analysts, warehousing engineers and data scientists to apply best practices.
• Act as data SME and guide governance standards.

Skills

SQL/NoSQL

DBMS skills

Python

Java

R

Scala

ETL tools

Data warehousing

Databricks

AWS

APIs

Distributed systems

Education

Postgraduate IT

Masters IT

Tools

Databricks

AWS

SQL Server

Job description

About the job Intermediate Data Engineer

• Post Graduate Degree: Information Technology
• Masters Degree: Information Technology

Experience

• Minimum 3 years experience
• Data Integrity
• IT Applications
• Data Analysis
• Knowledge Classification
• C++.
• Amazon Web Services.
• Amazon S3.
• Databricks

Skills:

• Database systems (SQL and NoSQL)
• Data engineer must know how to manipulate database management systems (DBMS), which is a software application that provides an interface to databases for information storage and retrieval
• Data warehousing solutions
• ETL tools
• ETL (Extract, Transfer, Load)
• Data APIs
• Allows two applications or machines to communicate with each other for a specified task
• Python, Java, R and Scala programming languages
• Understanding the basics of distributed systems

Responsibilities:

• Identify shortcomings and suggest improvements to existing processes, systems and procedures, then delivers a plan for a small element of a change management program with guidance from a project/program manager
• Build analytics tools that utilise the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics
• Monitor the existing metrics, analyse data, and lead partnership with other Data and Analytics teams in an effort to identify and implement system and process improvements
• Utilise data to discover tasks that can be automated and identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc
• Developing ETL processes that convert data into formats for consumption
• Risk, Regulatory, Prudential and Compliance
• Responsible for executing testing and validation in line with data governance and quality business requirements
• Liaise with and collaborate with data analysts, data warehousing engineers, and data scientists in finding and applying best practices within the Data and Analytics department as well as defining the business data requirements, which will ensure that the collected data is of high quality and optimal for use across the department and the business at large
• Acts as a subject matter expert from a data perspective and provides input into all decisions relating to data engineering and the use thereof.
• Provide guidance in terms of setting, governance standards