DATA ENGINEER

Competent Candidates; Randburg; On-site; ZAR 480,000 - 780,000; Full time

Job summary

Competent Candidates seeks a Data Engineer to join a well-established group of companies in Gauteng, focusing on analytics, data quality, and scalable pipelines. You will build ETL processes, create APIs for data access, and enforce multi-tenant data isolation following security guidelines.

The role requires at least 2 years in data engineering, a relevant degree, and hands-on experience with Python, SQL, data warehousing, and cloud platforms.

Qualifications

• Degree in Data Science, Information Technology, Computer Science or equivalent.
• Minimum of 2 years in a data engineer role or a similar technical role.

Responsibilities

• Build and maintain ETL pipelines supporting a multi-tenant data platform, ingesting data from APIs, databases, and event sources.
• Build and maintain Data Platform APIs that allow teams to ingest, process, and access data easily and reliably.
• Implemented tenant-specific logic by following existing configuration and naming conventions.
• Apply tenant-level data isolation using schemas, partitions, or access controls.
• Build models from existing templates used for financial and operational reporting. Develop models for analytics and reporting, maintaining consistency with shared data models.
• Monitor scheduled pipelines, investigate failures, and resolve data quality issues and inconsistencies.
• Maintain daily and incremental data loads into the data warehouse.
• Assist with onboarding new clients by validating source data and testing pipeline outputs.
• Work closely with senior data engineers to learn patterns for multi-tenant data isolation.
• Collaborate with analytics, product, and customer facing teams to understand reporting needs
• Support strict regulatory and audit requirements by following data handling, retention, and audit guidelines.
• Handle financial and sensitive data (PII) according to company policies and regulatory standards (e.g. POPIA).
• Apply least-privilege access and role-based access controls, and support data protection through masking, encryption, and established security standards.

Skills

Python

SQL

Data Warehouse

Databases

Cloud computing

Data modelling

ETL

Data Pipeline Development

CI/CD

Version Control

Education

Degree in Data Science/IT/Computer Science

Tools

Python

SQL

Job description

Our client, a well-established group of companies, is looking for a Data Engineer with strong analytics skills (including problem-solving, understanding data characteristics, identifying patterns, and addressing data quality issues). This is the ideal position for someone who wants to build their career with a company with an excellent reputation.

Formal Education

• Degree in Data Science, Information Technology, Computer Science or equivalent.

Advantageous

• Cloud Data Certifications.
• Exposure to regulated environments, financial services, and fintech.

Experience

• Minimum of 2 years in a data engineer role or a similar technical role.

Responsibilities

• Build and maintain ETL pipelines supporting a multi-tenant data platform, ingesting data from APIs, databases, and event sources.
• Build and maintain Data Platform APIs that allow teams to ingest, process, and access data easily and reliably.
• Implemented tenant-specific logic by following existing configuration and naming conventions.
• Apply tenant-level data isolation using schemas, partitions, or access controls.
• Build models from existing templates used for financial and operational reporting. Develop models for analytics and reporting, maintaining consistency with shared data models.
• Monitor scheduled pipelines, investigate failures, and resolve data quality issues and inconsistencies.
• Maintain daily and incremental data loads into the data warehouse.
• Assist with onboarding new clients by validating source data and testing pipeline outputs.
• Work closely with senior data engineers to learn patterns for multi-tenant data isolation.
• Collaborate with analytics, product, and customer facing teams to understand reporting needs
• Support strict regulatory and audit requirements by following data handling, retention, and audit guidelines.
• Handle financial and sensitive data (PII) according to company policies and regulatory standards (e.g. POPIA).
• Apply least-privilege access and role-based access controls, and support data protection through masking, encryption, and established security standards.

Technical Skills

• Programming languages - Good knowledge of programming languages such as Python, especially used for pipeline and data manipulation.
• SQL - working experience using SQL for data cleaning, aggregation, data transformation and integration.
• Data Warehouse - have a fundamental understanding of data warehousing solutions and platforms.
• Databases - hands-on experience working with relational and non-relational databases.
• Cloud computing - comfortable building data solutions using cloud-hosted services or data platforms.
• Data modelling and ETL - Can communicate and translate business requirements into existing data models.
• Data Pipeline Development - build and validate smaller-scale data pipelines independently.
• CI/CD and Version Control - apply best practices for managing pipelines and data workflows.

Core Cloud Data Technology, CI/CD and Programming tools