Data Engineer

Careerbox; Umhlanga Rocks; On-site; ZAR 600,000 - 900,000; Full time

Job summary

Careerbox is seeking a Data Engineer to design, expand and optimize our data and data pipeline architecture, enabling reliable data delivery for cross-functional teams including software developers, data analysts and data scientists. You will also collaborate with data warehouse managers and business units to map data between source systems and data marts, ensuring consistency across regions.

The role focuses on ETL development using SQL and Azure big data technologies, securing data delivery

Qualifications

• 3 years of experience writing queries, extracting data, data scrubbing, and managing large volumes of records.
• Experience with Relational and/or NoSQL databases is essential.
• Experience with ETL processes is essential.
• A qualification in Data Informatics or at least 3 years of relevant experience in large data environments.
• Experience managing a team of analysts and demonstrable technical expertise in writing complex database queries.

Responsibilities

• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional/non-functional business requirements.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from diverse sources using SQL and Azure big data technologies.
• Collaborate with stakeholders to assist with data-related technical issues and support data infrastructure needs.
• Create data tools for analytics and data science teams.
• Map data between source systems, data warehouses, and data marts.
• Design and execute reporting systems for processes.
• Research and implement technologies to ensure the data warehouse operates optimally.

Skills

SQL

Data extraction

Data scrubbing

Data modeling

Education

Data Informatics qualification

Tools

Azure big data technologies

Relational databases

NoSQL databases

ETL processes

Job description

Data Engineer Job Description

The data engineer is responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The Data Engineer will support software developers, database architects, data warehouse managers, data analysts, and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.

Duties & Responsibilities

Key Responsibilities:

• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional/non-functional business requirements.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure ‘big data’ technologies.
• Work with stakeholders including the Executive, Service, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Create data tools for analytics and data science teams.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Map data between source systems, data warehouses, and data marts.
• Keep our data separated and secure across national boundaries through multiple data centers and Cloud regions.
• Implement business rules via stored procedures, middleware, or other technologies.
• Develop and implement data extraction procedures from other systems, such as call centers, HR, or external systems.
• Create supporting documentation, such as metadata and diagrams of entity relationships, business processes, and process flow.
• Create plans, test files, and scripts for database testing, ranging from unit to integration testing.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, redesigning infrastructure for greater scalability, etc.
• Select methods, techniques, or criteria for database evaluative procedures.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Design and execute reporting systems for processes.
• Analyze system requirements and gather all business information.
• Research and implement appropriate technologies to ensure that the data warehouse is always operating optimally.

Desired Experience & Qualification

Experience, Knowledge and Qualifications:

• The ideal candidate is currently a Database Administrator with 3 years of experience in writing queries, extracting data, data scrubbing, and managing large volumes of records.
• Previous experience with Relational and/or NoSQL databases is essential.
• Previous experience with ETL processes is essential.
• A qualification in Data Informatics or at least 3 years of relevant experience in large data environments.
• Experience managing a team of analysts and demonstrable technical expertise in writing complex database queries.