Data Engineer: Build Scalable AI Data Pipelines

DataProphet; Cape Town; On-site; ZAR 720,000 - 1,100,000; Full time

Job summary

DataProphet seeks a Data Engineer to design, build and maintain data infrastructure and pipelines enabling AI solutions. You will handle diverse data sources to ensure reliable, accessible data for scientists and ML systems.

You will own data pipelines end-to-end from ingestion to delivery, build clean datasets, and contribute to data models and lakehouse structures, with a focus on quality, monitoring and operational excellence.

Qualifications

• Production data infrastructure experience required.
• End-to-end ownership of data pipelines from ingestion to delivery.
• Experience with data quality, monitoring and incident response.
• Certifications in cloud or data platforms are a plus.

Responsibilities

• Design, build, test and maintain scalable production data pipelines & infrastructure.
• Own data pipelines end-to-end across ingestion, transformation, storage and delivery.
• Integrate data from varied source systems into DataProphet's data environment.
• Build reliable, clean and usable datasets for Data Scientists and ML systems.
• Design and maintain data models and data warehouse/lakehouse structures.
• Implement data quality validation, monitoring and alerting.
• Diagnose and resolve pipeline failures and data-quality issues.
• Build prototypes and PoC solutions as needed.
• Evaluate hardware, software and cloud solutions for data systems.
• Contribute to data and system architecture and design.
• Integrate new data management tools and create analytics components.
• Install and update disaster recovery procedures.

Skills

SQL

Python

Data modelling

Data warehousing

Airflow

Apache Spark

Git

Databricks

Snowflake

Terraform

Kafka

Kinesis

Cloud platforms

CI/CD

System design

Education

Bachelor's / Honours / Master’s / PhD in Computer Science or related field

Tools

Airflow

Dagster

Apache Spark

Prefect

Git

Databricks

ClickHouse

Snowflake

DuckDB

Terraform

Job description

DataProphet seeks a Data Engineer to design, build and maintain data infrastructure and pipelines enabling AI solutions. You will handle diverse data sources to ensure reliable, accessible data for scientists and ML systems.

You will own data pipelines end-to-end from ingestion to delivery, build clean datasets, and contribute to data models and lakehouse structures, with a focus on quality, monitoring and operational excellence.