Data Engineer

Creative Leadership Solutions; Pretoria; On-site; ZAR 400 000 - 600 000; Full time

Job summary

A prominent leadership consulting firm is looking for a skilled ETL Developer to design and implement efficient data pipelines and manage cloud data solutions using Azure technologies. The ideal candidate will have a BSc in Computer Science or Engineering and 2-4 years of experience in data transformation and cloud-based data processing. You will be responsible for the entire ETL process and ensuring optimal data architecture and performance across systems. Join a dynamic team dedicated to leveraging data for strategic insights.

Qualifications

• 2-4 years of experience in relevant fields.
• Hands-on experience developing ETL pipelines and data transformation.
• Familiar with cloud-based data processing using Azure tools.

Responsibilities

• Design and orchestrate ETL pipelines for data ingestion.
• Implement robust data transformation logic using Python.
• Architect data warehouses and manage data architectures.

Skills

SQL Server

Azure Synapse Analytics

Azure Blob Storage

Python

ETL processes

Data Factory

Dimensional Modelling

Power BI

C#

.NET

JavaScript

HTML/CSS

Education

BSc in Computer Science, Engineering or relevant field

Job description

REQUIREMENTS

Minimum education (essential) : BSc in Computer Science, Engineering or relevant field

Minimum applicable experience (years) : 2-4 years

Required nature of experience : Experience with SQL Server and Azure Synapse Analytics / Microsoft Fabric for query writing, indexing, performance tuning and schema design.

Hands‑on experience developing ETL pipelines, including data extraction from REST / SOAP APIs, databases and flat files.

Proficiency in data transformation using Python and Azure-native tools.

Experience with data warehousing. Background in data modelling, including dimensional modelling, schema evolution and versioning.

Practical knowledge of cloud-based data storage and processing using Azure Blob Storage.

Familiarity with pipeline optimisation, fault tolerance, monitoring and security best practices.

Experience developing web applications using C# and the .

NET platform. Experience with front‑end development using Blazor, React.js, JavaScript / TypeScript, HTML, CSS / SCSS.

Skills and Knowledge (essential)

• SQL Server, Azure Synapse Analytics, Azure Blob Storage, Microsoft Fabric
• Python
• REST / SOAP APIs, Data Extraction, Transformation, Loading (ETL)
• Azure Data Factory, Pipeline Orchestration
• Dimensional Modelling, Schema Evolution, Data Warehousing
• Power BI
• Performance Optimisation, Indexing, Query Tuning
• Cloud Data Processing, Backups
• C#, .
• NET, Blazor
• JavaScript / TypeScript, HTML, CSS / SCS

Other

• Proficient in Afrikaans and English
• Own transport and license

KEY PERFORMANCE AREAS AND OBJECTIVES

ETL and Pipeline Development

• Design, build, and orchestrate efficient ETL pipelines using Azure Synapse for both batch and near‑real‑time data ingestion.
• Extract data from a variety of structured and unstructured sources including REST APIs, SOAP APIs, databases, and flat files.
• Apply robust data transformation logic using Python and native Azure Synapse transformation tools.
• Optimise data flows for performance, scalability, and cost‑effectiveness.
• Implement retry mechanisms, logging and monitoring within pipelines to ensure data integrity and fault tolerance.

Data Architecture and Management

• Design and manage scalable and efficient data architectures using Microsoft SQL Server and Azure services, including Synapse Analytics / Microsoft Fabric and Blob Storage.
• Develop robust schema designs, indexes and query strategies to support analytical and operational workloads.
• Support schema evolution and version control, ensuring long‑term maintainability and consistency across datasets.
• Implement and maintain metadata repositories and data dictionaries for improved data governance and transparency.
• Define and maintain role‑based access control to ensure data security and compliance.

Data Warehousing and BI Integration

• Architect and manage enterprise data warehouses using Azure Synapse Analytics.
• Apply best practices for data loading, partitioning strategies, and storage optimisation.
• Integrate warehousing solutions with Power BI and other analytics platforms for seamless business intelligence consumption.

Data Modelling & Standards

• Develop and maintain conceptual, logical and physical data models.
• Implement dimensional modelling techniques (e.g., star / snowflake schemas) to support advanced analytics and reporting.
• Apply normalisation standards and relational modelling techniques to support OLTP and OLAP workloads.
• Ensure consistency of data models across systems and support schema versioning and evolution.

Reporting and Communication

• Provide clear, timely updates on task status and progress to senior developers / management.
• Contribute to reports, manuals and other documentation related to software status, operation, and maintenance.
• Collaborate effectively with team members and stakeholders using the appropriate communication channels.
• Maintain system and product change logs and release notes according to company standards.

Automation, Monitoring and Optimisation

• Automate recurring data engineering tasks and deploy solutions with CI / CD best practices.
• Implement monitoring and alerting mechanisms to detect data quality issues and pipeline failures.
• Analyse and optimise query performance across platforms (SQL Server, Azure Synapse).
• Support scalability planning and cost control by monitoring pipeline execution and resource usage.

Security and Best Practices

• Enforce security best practices for data access, including encryption and secure authentication.
• Ensure compliance with data governance policies and applicable regulatory standards.
• Document processes, architectural decisions and technical implementations in alignment with organisational standards.

Contribution to The Team

• Collaborate with developers, data analysts, data scientists and business teams to understand data requirements and deliver scalable solutions.
• Work with the team to integrate pipelines with source control and deployment workflows.

Quality Management and Compliance

• Document data processes, transformations and architectural decisions.
• Maintain high standards of software quality within the team by adhering to good processes, practices and habits.
• Ensure compliance to the established processes and standards for the development lifecycle, including but not limited to data archival.
• Safeguard confidential information and data.