Senior Azure Data Platform Engineer for AI & Analytics

Cf Consulting; Gauteng; On-site; ZAR 700,000 - 1,000,000; Full time

Benefits offered by this job

Competitive salary

On-site work environment

Medical, dental, and vision insurance

Professional development opportunities

Certifications

Job summary

Cf Consulting in Centurion is seeking a meticulous Senior Data Engineer to build and optimize scalable data pipelines for AI initiatives, leveraging Azure Fabric and related data platforms. You will ensure availability, quality, and accessibility of data for ML models and analytics.

You will work with data scientists and BI teams to architect solutions, maintain data quality, and support regulatory compliance in a fast-paced environment. On-site role in Centurion with a collaborative team.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Data Engineering or a related field.
• Strong hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
• Strong SQL Server and T-SQL capability, including complex query development, schema design, indexing and performance optimisation.
• Practical experience developing, maintaining and supporting production ETL/ELT pipelines.
• Experience integrating and extracting data from REST/SOAP APIs, databases, flat files and other structured or unstructured data sources.
• Proficiency in data transformation using SQL and Python/PySpark, with an understanding of scalable data processing practices.
• Practical experience in data warehousing, dimensional modelling, incremental loading, orchestration and schema evolution.
• Experience with source control, CI/CD and deployment practices using Git, Azure DevOps or equivalent tools.
• Experience supporting Power BI and other downstream analytical or reporting requirements.
• Demonstrated ability to take ownership of an existing technical environment with limited hand-holding.

Responsibilities

• Design, construct, install, test, and maintain highly scalable data management systems and pipelines.
• Develop processes and systems to monitor data quality, ensure consistency, and manage data flow.
• Build and manage ETL/ELT processes for large, diverse datasets.
• Optimize data infrastructure for performance, reliability, and cost-effectiveness.
• Collaborate with data scientists and ML engineers to understand their data requirements and provide solutions.
• Implement data security and governance best practices.

Skills

Python

SQL

PySpark

ETL/ELT

Data Modeling

Data Warehousing

Azure Data Platform

Power BI

REST/SOAP APIs

CI/CD

Git

Education

Bachelor's or Master's degree in Computer Science / Engineering / related field

Tools

Microsoft Fabric

Azure Data Factory

OneLake

Lakehouse

Data Pipelines/Data Factory

Dataflows Gen2

Git/Azure DevOps

Python

Job description

Cf Consulting in Centurion is seeking a meticulous Senior Data Engineer to build and optimize scalable data pipelines for AI initiatives, leveraging Azure Fabric and related data platforms. You will ensure availability, quality, and accessibility of data for ML models and analytics.

You will work with data scientists and BI teams to architect solutions, maintain data quality, and support regulatory compliance in a fast-paced environment. On-site role in Centurion with a collaborative team.