Data Engineering Lead – Data Science and AI

DVT; Cape Town; On-site; ZAR 1,400,000 - 2,200,000; Full time

Job summary

DVT is seeking an experienced Data Engineering Lead to design, build and govern data and architecture for AI initiatives. You will own Azure- and AWS-based environments, develop governed data pipelines and collaborate with data scientists and engineers to productionise AI capabilities.

You will lead the data engineering team, ensure data quality and governance, and drive cost optimisation and incident management while aligning with security standards.

Qualifications

• 7+ years of data engineering, architecture and cloud platform experience.
• predecessor
• Needs strong cloud alignment and governance understanding.
• Azure and/or AWS certifications are a plus.
• Experience with Azure data & AI services and AWS/Azure data integration.
• Proven data pipelines: batch, streaming, API-based.
• Experience enabling Data Science/AI with MLOps capabilities.
• Strong Python/SQL, data modelling and transformations.
• Experience with Git-based CI/CD and IaC.
• Knowledge of data governance and enterprise architecture coordination.

Responsibilities

• Define and maintain target data engineering architecture for Data Science and AI workloads.
• Design patterns for moving data from AWS to Azure Data Science env; collaborate with IT & security teams.
• Own data engineering delivery: pipelines, transformations and curated datasets.
• Define reusable features, feature stores, lineage, versioning and monitoring.
• Establish production-grade MLOps/LLMOps: model deployment, versioning and retraining.
• Provide data engineering foundations for Generative/agentic AI: ingestion, indexing and governance.
• Ensure compliance with governance, privacy and security standards.
• Lead Data Engineers; translate roadmap into an engineering plan.
• Establish monitoring, support, incident management and cost optimisation; liaison with infra & governance teams.

Skills

Senior data engineering

Architecture

Python

SQL

MLOps

LLMOps

Data governance

Education

Degree in CS/IS/Data Eng/Math

Tools

Azure Data Factory

Azure Machine Learning

Azure Functions

Azure Storage

AWS (S3, Glue, Redshift)

Git CI/CD

Terraform

Databricks

Job description

DVT is a leading technology consulting and software engineering company delivering innovative solutions across Africa and internationally. We partner with clients to solve complex business challenges through software engineering, cloud platforms, data, artificial intelligence, and digital transformation. Our teams combine deep technical expertise with a strong consulting mindset to create measurable business value.

Our Data Practice is looking for an experienced Data Engineering Lead to design, build and govern the data and technology architecture underpinning our Data Science, Machine Learning, Generative AI and agentic AI initiatives. You will provide the engineering capability required to move analytical and AI solutions from experimentation into secure, scalable, production-ready services.

This hands-on technical leadership role combines architecture ownership, engineering delivery and coordination of engineering resources, with a core focus on establishing the Azure-based architecture used by the team, including governed data pipelines between our AWS data environment and Azure-based Data Science and AI platforms.

Key Responsibilities:

• Define and maintain the target data engineering architecture for Data Science and AI workloads, including data preparation, model deployment, GenAI applications and agentic workflows.

• Design the architecture and integration patterns for bringing data from AWS into the Azure Data Science environment, working with IT, Data Platform, Cloud and Security teams.

• Own data engineering delivery, building reusable pipelines, transformations and curated datasets, and standardised frameworks for ingestion, transformation, orchestration and publishing.

• Define the approach to reusable features and data products, including feature-store capabilities, lineage, versioning and monitoring.

• Partner with Data Scientists and Engineers to establish production-grade MLOps and LLMOps capabilities, including model deployment, versioning, monitoring and retraining.

• Provide the data engineering foundations for Generative AI and agentic AI solutions, including ingestion, chunking, indexing and governance of enterprise information.

• Ensure data and AI solutions align with governance, privacy, responsible AI, security and data quality standards.

• Provide technical leadership to Data Engineers supporting the Data Science portfolio, and translate the roadmap into a prioritised engineering and architecture plan.

• Establish monitoring, operational support, incident management and cost-optimisation practices, and act as primary technical liaison with architecture, infrastructure, cybersecurity and governance teams.

Essential Experience & Qualifications Required

• Proven, senior-level experience (7+ years) in data engineering, architecture, and cloud platforms.

• Degree in Computer Science, Information Systems, Data Engineering, Software Engineering, Mathematics or a related field.

• Microsoft Azure and/or AWS architecture or engineering certifications advantageous.

• Strong practical experience with Microsoft Azure data and AI services (e.g. Azure Data Factory, Azure Functions, Azure Storage, Azure Machine Learning), and experience integrating data across AWS and Azure environments.

• Experience building and operating production data pipelines (batch, incremental, streaming or API-based), with data orchestration, automated testing, monitoring and deployment practices.

• Experience supporting Data Science, machine learning or AI teams, including implementing MLOps or model deployment capabilities.

• Strong Python and SQL skills, with experience in data modelling and transformation frameworks.

• Experience with Git-based source control, CI/CD pipelines and Infrastructure as Code.

• Experience working with data governance, information security and enterprise architecture teams, and leading technical delivery or coordinating engineers.

Advantageous Experience

• Experience with Microsoft Fabric, Synapse Analytics or Databricks.

• Experience with AWS services such as S3, Glue, Athena, Redshift, Lambda or SageMaker.

• Experience building or supporting feature stores, model registries and experiment-tracking platforms.

• Experience with containerisation, orchestration technologies, and relational, document, vector or analytical databases.

• Exposure to FinOps and cloud cost optimisation practices.

Important to Know

This is a hands-on Data Engineering and Cloud Architecture leadership role embedded within the Data Science and AI function, not a pure infrastructure or platform-operations role. The focus is establishing the data architecture, pipelines and engineering foundations that enable Data Science, ML and AI delivery.