Senior Data Engineer — Azure Data Platform & Analytics

Cf Consulting; Gauteng; On-site; ZAR 700,000 - 1,000,000; Full time

Benefits offered by this job

Competitive salary

On-site work environment

Medical, dental, and vision insurance

Professional development opportunities

Certifications

Job summary

Cf Consulting in Centurion is seeking a meticulous Senior Data Engineer to build and optimize scalable data pipelines for AI initiatives, leveraging Azure Fabric and related data platforms. You will ensure availability, quality, and accessibility of data for ML models and analytics.

You will work with data scientists and BI teams to architect solutions, maintain data quality, and support regulatory compliance in a fast-paced environment. On-site role in Centurion with a collaborative team.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Data Engineering or a related field.
• Strong hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
• Strong SQL Server and T-SQL capability, including complex query development, schema design, indexing and performance optimisation.
• Practical experience developing, maintaining and supporting production ETL/ELT pipelines.
• Experience integrating and extracting data from REST/SOAP APIs, databases, flat files and other structured or unstructured data sources.
• Proficiency in data transformation using SQL and Python/PySpark, with an understanding of scalable data processing practices.
• Practical experience in data warehousing, dimensional modelling, incremental loading, orchestration and schema evolution.
• Experience with source control, CI/CD and deployment practices using Git, Azure DevOps or equivalent tools.
• Experience supporting Power BI and other downstream analytical or reporting requirements.
• Demonstrated ability to take ownership of an existing technical environment with limited hand-holding.

Responsibilities

• Design, construct, install, test, and maintain highly scalable data management systems and pipelines.
• Develop processes and systems to monitor data quality, ensure consistency, and manage data flow.
• Build and manage ETL/ELT processes for large, diverse datasets.
• Optimize data infrastructure for performance, reliability, and cost-effectiveness.
• Collaborate with data scientists and ML engineers to understand their data requirements and provide solutions.
• Implement data security and governance best practices.

Skills

Python

SQL

PySpark

ETL/ELT

Data Modeling

Data Warehousing

Azure Data Platform

Power BI

REST/SOAP APIs

CI/CD

Git

Education

Bachelor's or Master's degree in Computer Science / Engineering / related field

Tools

Microsoft Fabric

Azure Data Factory

OneLake

Lakehouse

Data Pipelines/Data Factory

Dataflows Gen2

Git/Azure DevOps

Python

Job description

• What Senior Data Engineer Azure Data Platform ...

Showing 17 Senior Data Engineer Azure Data Platform Analytics jobs in Centurion

An established insurance company is seeking to hire a highly skilled and experienced Data Engineer to join their team. Your:

Formal Education:

• Degree in Data Science, Information Technology, Computer Science or equivalent

Advantageous :

• Cloud Data Certifications
• Exposure to regulated environments,financial services, fintech

Experience:

• Minimum of 2 years in a data engineer role or a similar technical role.

Responsibilities:

• Build and maintain ETL pipelines supporting a multi-tenant data platform, ingesting data from APIs, databases, and event sources.
• Build and maintain Data Platform APIs that allow teams to ingest, process, and access data easily and reliably.
• Implemented tenant-specific logic by following existing configuration and naming conventions.
• Apply tenant-level data isolation using schemas, partitions, or access controls
• Build models from existing templates used for financial and operational reporting. Develop models for analytics and reporting, maintaining consistency with shared data models.
• Monitor scheduled pipelines, investigate failures, and resolve data quality issues and inconsistencies.
• Maintain daily and incremental data loads into the data warehouse.
• Assist with onboarding new clients by validating source data and testing pipeline outputs.
• Work closely with senior data engineers to learn patterns for multi-tenant data isolation.
• Collaborate with analytics, product, and customer facing teams to understand reporting needs.
• Support strict regulatory and audit requirements by following data handling,retention, and audit guidelines.
• Handle financial and sensitive data (PII) according to company policies and regulatory standards (e.g. POPIA)
• Apply least-privilege access and rolebased access controls, and support data protection through masking, encryption,and established security standards.

Technical Skills:

• Programming languages Good knowledge of programming languages such as Python, especially used for pipeline and data manipulation.
• SQL working experience using SQL for data cleaning, aggregation, data transformation and integration.
• Data Warehouse - have a fundamental understanding of data warehousing solutions and platforms.
• Databases hands on experience working with relational and non-relational databases.
• Cloud computing comfortable building data solutions using cloud hosted services or data platforms.
• Analytics skills strong problem solving skills, understand data characteristics, identify patterns, and data quality issues.
• Data modelling and ETL Is able to communicate and translate business requirements into existing data models.
• Data Pipeline Development build and validate smaller scale data pipelines independently.
• CI/CD and Version Control apply best practice for managing pipelines and data workflows.
• Bachelor's degree in Computer Science, Information Systems, Data Engineering or a related field.
• Relevant Microsoft Fabric and/or Azure data certification.

REQUIREMENTS Minimum Education (essential)

• Bachelor's degree in Computer Science, Information Systems, Data Engineering or a related field.
• Relevant Microsoft Fabric and/or Azure data certification.

Minimum applicable experience (years):

• 0-1 years of practical data engineering experience, including strong recent hands-on experience with Microsoft Fabric.

Required nature of experience:

• Strong hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
• Strong SQL Server and T-SQL capability, including complex query development, schema design, indexing and performance optimisation.
• Practical experience developing, maintaining and supporting production ETL/ELT pipelines.
• Experience integrating and extracting data from REST/SOAP APIs, databases, flat files and other structured or unstructured data sources.
• Proficiency in data transformation using SQL and Python/PySpark, with an understanding of scalable data processing practices.
• Practical experience in data warehousing, dimensional modelling, incremental loading, orchestration and schema evolution.
• Experience with troubleshooting pipeline failures, data quality issues and performance bottlenecks, including the ability to restore service efficiently.
• Experience with source control, CI/CD and deployment practices using Git, Azure DevOps or equivalent tools.
• Experience supporting Power BI and other downstream analytical or reporting requirements.
• Demonstrated ability to take ownership of an existing technical environment with limited hand-holding.
• Strong documentation, communication and stakeholder engagement skills.

Skills and Knowledge (essential):

• Microsoft Fabric: OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
• SQL Server / T-SQL
• Python / PySpark
• REST/SOAP APIs and structured/unstructured data ingestion.
• ETL/ELT, incremental loading, orchestration and scheduling.
• Dimensional modelling, medallion architecture, schema evolution and data warehousing.
• Power BI integration and understanding of downstream analytical requirements.
• Git / Azure DevOps, CI/CD and environment deployment practices.
• Monitoring, data quality, performance optimisation, security and operational support.

Other:

• Proficient in Afrikaans and English.
• Own transport and valid drivers license.

Remuneration Offered Market related

Our client is seeking a meticulous and experienced Data Engineer to build and optimize robust data pipelines for AI & Emerging Technologies initiatives in Centurion . In this critical role, you will be responsible for ensuring the availability, quality, and accessibility of data required for machine learning models and AI-driven applications. You will work closely with data scientists and ML engineers to architect scalable data solutions, fostering a data-centric culture. This on-site position provides a unique opportunity to work with a dedicated team in a collaborative and innovative environment.

About the Role

Our client is seeking a meticulous and experienced Data Engineer to build and optimize robust data pipelines for AI & Emerging Technologies initiatives in Centurion . In this critical role, you will be responsible for ensuring the availability, quality, and accessibility of data required for machine learning models and AI-driven applications. You will work closely with data scientists and ML engineers to architect scalable data solutions, fostering a data-centric culture. This on-site position provides a unique opportunity to work with a dedicated team in a collaborative and innovative environment.

Key Responsibilities

• Design, construct, install, test, and maintain highly scalable data management systems and pipelines.
• Develop processes and systems to monitor data quality, ensure consistency, and manage data flow.
• Build and manage ETL/ELT processes for large, diverse datasets.
• Optimize data infrastructure for performance, reliability, and cost-effectiveness.
• Collaborate with data scientists and ML engineers to understand their data requirements and provide solutions.
• Implement data security and governance best practices.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 3+ years of experience in data engineering, with a focus on building large-scale data pipelines.
• Proficiency in SQL and programming languages like Python or Scala.
• Experience with big data technologies such as Spark, Hadoop, Kafka, and distributed storage systems.
• Hands-on experience with cloud data services (AWS, Azure, GCP).
• Strong understanding of data warehousing concepts and database design.

Benefits

• Competitive annual salary and performance incentives.
• On-site work environment with access to advanced facilities.
• Comprehensive medical, dental, and vision insurance plans.
• Opportunities for professional development and certifications.
• Collaborative team atmosphere focused on innovation and growth.