Data Engineering Lead – Data Science and AI

dvtcareers; Johannesburg; On-site; ZAR 1,200,000 - 2,000,000; Full time

Job summary

DVT, a leading technology consulting and software engineering company, seeks a Data Engineering Lead in Johannesburg to design, build and govern data architectures for Data Science, ML, and AI initiatives. You will own end-to-end data pipelines spanning AWS and Azure environments, with a hands-on leadership approach.

You will collaborate with Data Scientists and Engineers to implement production-ready MLOps and GenAI workflows, ensuring governance, security and data quality across enterprise

Qualifications

• Senior-level experience (7+ years) in data engineering, architecture and cloud platforms.
• Microsoft Azure and/or AWS architecture or engineering certifications are advantageous.
• Strong Python and SQL skills with data modelling and transformation experience.
• Experience building and operating production data pipelines with orchestration, testing, monitoring and deployment.

Responsibilities

• Define and maintain the target data engineering architecture for Data Science and AI workloads.
• Design the integration patterns for bringing data from AWS into the Azure Data Science environment.
• Own data engineering delivery with reusable pipelines, transformations and curated datasets.
• Establish production-grade MLOps and LLMOps capabilities including model deployment and monitoring.
• Provide data engineering foundations for Generative AI and agentic AI solutions.
• Ensure solutions comply with governance, privacy, security and data quality standards.
• Lead Data Engineers and translate roadmaps into prioritised engineering plans.
• Establish monitoring, incident management and cost-optimisation practices.

Skills

Data engineering

Cloud architecture

Azure

AWS

Python

SQL

CI/CD

Infrastructure as Code

Education

Degree in Computer Science/Information Systems/Data Engineering

Tools

Azure Data Factory

Azure Functions

Azure Storage

Azure Machine Learning

AWS S3

AWS Glue

Git

CI/CD pipelines

Infrastructure as Code

Job description

DVT is a leading technology consulting and software engineering company delivering innovative solutions across Africa and internationally. We partner with clients to solve complex business challenges through software engineering, cloud platforms, data, artificial intelligence, and digital transformation. Our teams combine deep technical expertise with a strong consulting mindset to create measurable business value.

Our Data Practice is looking for an experienced Data Engineering Lead to design, build and govern the data and technology architecture underpinning our Data Science, Machine Learning, Generative AI and agentic AI initiatives. You will provide the engineering capability required to move analytical and AI solutions from experimentation into secure, scalable, production-ready services.

This hands-on technical leadership role combines architecture ownership, engineering delivery and coordination of engineering resources, with a core focus on establishing the Azure-based architecture used by the team, including governed data pipelines between our AWS data environment and Azure-based Data Science and AI platforms.

Key Responsibilities:

• Define and maintain the target data engineering architecture for Data Science and AI workloads, including data preparation, model deployment, GenAI applications and agentic workflows.
• Design the architecture and integration patterns for bringing data from AWS into the Azure Data Science environment, working with IT, Data Platform, Cloud and Security teams.
• Own data engineering delivery, building reusable pipelines, transformations and curated datasets, and standardised frameworks for ingestion, transformation, orchestration and publishing.
• Define the approach to reusable features and data products, including feature-store capabilities, lineage, versioning and monitoring.
• Partner with Data Scientists and Engineers to establish production-grade MLOps and LLMOps capabilities, including model deployment, versioning, monitoring and retraining.
• Provide the data engineering foundations for Generative AI and agentic AI solutions, including ingestion, chunking, indexing and governance of enterprise information.
• Ensure data and AI solutions align with governance, privacy, responsible AI, security and data quality standards.
• Provide technical leadership to Data Engineers supporting the Data Science portfolio, and translate the roadmap into a prioritised engineering and architecture plan.
• Establish monitoring, operational support, incident management and cost-optimisation practices, and act as primary technical liaison with architecture, infrastructure, cybersecurity and governance teams.

Essential Experience & Qualifications Required

• Proven, senior-level experience (7+ years) in data engineering, architecture, and cloud platforms.
• Degree in Computer Science, Information Systems, Data Engineering, Software Engineering, Mathematics or a related field.
• Microsoft Azure and/or AWS architecture or engineering certifications advantageous.
• Strong practical experience with Microsoft Azure data and AI services (e.g. Azure Data Factory, Azure Functions, Azure Storage, Azure Machine Learning), and experience integrating data across AWS and Azure environments.
• Experience building and operating production data pipelines (batch, incremental, streaming or API-based), with data orchestration, automated testing, monitoring and deployment practices.
• Experience supporting Data Science, machine learning or AI teams, including implementing MLOps or model deployment capabilities.
• Strong Python and SQL skills, with experience in data modelling and transformation frameworks.
• Experience with Git-based source control, CI/CD pipelines and Infrastructure as Code.
• Experience working with data governance, information security and enterprise architecture teams, and leading technical delivery or coordinating engineers.

Advantageous Experience

• Experience with Microsoft Fabric, Synapse Analytics or Databricks.
• Experience with AWS services such as S3, Glue, Athena, Redshift, Lambda or SageMaker.
• Experience building or supporting feature stores, model registries and experiment-tracking platforms.
• Experience with containerisation, orchestration technologies, and relational, document, vector or analytical databases.
• Exposure to FinOps and cloud cost optimisation practices.

Important to Know

This is a hands-on Data Engineering and Cloud Architecture leadership role embedded within the Data Science and AI function, not a pure infrastructure or platform-operations role. The focus is establishing the data architecture, pipelines and engineering foundations that enable Data Science, ML and AI delivery.