Data Engineer

Shop2Shop

Stellenbosch

Híbrido

ZAR 600.000 - 1.000.000

Jornada completa

14 días+

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

Shop2Shop in Stellenbosch, South Africa, seeks a Data Engineer to design, build and maintain a modern data platform using Python and Dagster, delivering reliable data pipelines and governance. You will work with data analysts, DevOps and business stakeholders to translate data requirements into scalable solutions, focusing on data quality, lineage and observability.

The role offers a hybrid working model with regular in-office collaboration in Stellenbosch, and opportunities to experiment with

Formación

  • 3–5 years of professional software or data engineering experience.
  • Experience building, deploying and supporting production data pipelines with Dagster or similar tool.
  • Strong Python programming and ETL/ELT design skills.
  • Experience with data modeling and cloud data warehouses.
  • Familiarity with data validation/testing (e.g., Pytest) and CI/CD pipelines.
  • Awareness of data privacy (POPIA) and data governance.

Responsabilidades

  • Design, build and maintain robust, declarative assets and graphs in Dagster.
  • Write clean Python to extract, transform and load data into the warehouse.
  • Implement data validation, testing and monitoring to ensure quality before reporting.
  • Own day-to-day pipeline operations, including failures, reruns and backfills.
  • Integrate Dagster with the broader data stack (dbt, DuckDB, Athena, PostgreSQL, Terraform).
  • Automate deployments using Docker, Kubernetes and GitLab CI.

Conocimientos

Python
Data engineering
Dagster
Data pipelines
Data quality
AI tools curiosity

Educación

Relevant IT qualification or equivalent practical experience

Herramientas

Dagster
Airflow
Prefect
dbt
Docker
Kubernetes
Terraform
Athena
Snowflake
BigQuery
Redshift
DuckDB

Descripción del empleo

The ability to make a payment quickly and at a low cost is an underestimated bedrock of our economy in South Africa. Payments are the financial lifeline of every business and individual. We are on a mission to help entrepreneurs participate and prosper in the digital economy by solving their payment challenges. When you join Shop2Shop, you join a team of people driven by a shared purpose for why we do what we do every day. The experience of collaboration across teams will be unrivalled. The feeling of belonging to a talented team of people will make this your last career stop. You will find a stimulating and challenging smorgasbord of problems to solve with the aim of supporting entrepreneurs in South Africa.

Purpose of role:

Every decision Shop2Shop makes about merchants, risk and growth depends on data arriving on time and being right. The Data Engineer builds and runs the pipelines that make that possible.

You will design, build and maintain Shop2Shop’s modern data platform, using Python and Dagster to develop data pipelines centred on software-defined assets. The role applies modern software engineering practices to data infrastructure, with a focus on testability, data lineage, declarative asset management and reliable data delivery.

Reporting to the Core Data Services Manager, you will work closely with data analysts, DevOps and business stakeholders to turn data requirements into scalable technical solutions.

Success = The people who rely on our data can trust it without checking it twice.

You’ll thrive in this role if…
  • You approach data infrastructure with a software engineering mindset and value clean, maintainable Python code.
  • You care about testability, data lineage and declarative asset management.
  • You take ownership of data quality, monitoring and the reliability of data pipelines.
  • You enjoy automating deployments and connecting tools across a modern data stack.
  • You work well with data analysts, DevOps teams and business stakeholders to translate data requirements into scalable solutions.
  • You are curious about how AI tools can make you faster and better, and you check what they produce.
What You’ll Do (and what success looks like)

Dagster Assets and Data Lineage

  • Design, build and maintain robust, declarative software-defined assets and graphs in Dagster
  • Use Dagster to track data lineage and state across the data stack

Success = Anyone can see where a number came from and how fresh it is.

  • Write clean, efficient and maintainable Python code to extract, transform and load data from disparate sources into the data warehouse
  • Use ETL or ELT processes as best fits each source
  • Build data models in line with the architecture and modelling standards set for the platform

Success = Data lands in the warehouse on time, every time, without manual fixes.

Data Quality and Observability

  • Implement rigorous data validation, testing and monitoring using tools such as Pytest and Dagster’s built-in features
  • Make sure data quality issues are detected and flagged before they reach reports and dashboards

Success = Data problems are caught by our checks before someone finds them in a dashboard.

Production Support

  • Own the day-to-day running of the pipelines you build, including failures, reruns and backfills
  • Investigate and resolve pipeline incidents, and fix the root cause so they do not recur
  • Keep runbooks and documentation current so others can support the platform

Success = When a pipeline fails, it is noticed, fixed and understood quickly, and the same failure does not happen twice.

Infrastructure and Integration

  • Connect Dagster with the broader modern data stack, including dlthub, dbt, DuckDB, Athena, PostgreSQL and Terraform
  • Integrate the data platform with AWS cloud infrastructure, working with DevOps, who own the underlying platform and clusters

Success = The data stack works as one platform rather than a set of separate tools.

CI/CD and Deployment

  • Automate the deployment of Dagster code and data infrastructure using Docker, Kubernetes and GitLab CI
  • Work within the platform and infrastructure standards set by DevOps

Success = Pipeline changes ship safely and repeatably, with no manual deployment steps.

Data Protection

  • Handle merchant, customer and transaction data in line with POPIA and Shop2Shop’s data governance standards
  • Apply the access controls, masking and retention rules defined in those standards
  • Flag data protection risks early

Success = Personal data is protected by how the pipelines are built, not by remembering to be careful.

Use AI to Improve Delivery

  • Use approved AI tools to speed up coding, testing and documentation
  • Review AI-generated code with the same rigour as any other code before it ships
  • Share what works with the Data team

Success = AI makes the team faster without lowering the quality bar.

  • Partner with data analysts, DevOps teams and business stakeholders to understand data requirements
  • Translate those requirements into scalable technical solutions

Success = Analysts and the business get data in the shape they need without long back and forth.

Core Competencies
  • Systems Thinking – Understands dependencies and sees the bigger picture
  • Problem Solving – Solves issues practically and quickly. Balances short-term fixes with long-term improvement.
  • Collaboration – Works well across technology teams. Supports shared success over silo thinking.
  • Continuous Improvement – Looks for better ways of working. Embraces automation, AI, and smarter delivery methods.
  • Curiosity and critical thinking – Asking “why” and “what if” before jumping to “how”
  • Communication – You translate technical issues into language business stakeholders understand.
Skills & Knowledge
  • Strong, idiomatic Python, including ETL and ELT pipeline design
  • Dagster concepts, including Software-Defined Assets, Ops, Graphs, Resources and I/O Managers, or the equivalent concepts in a similar orchestration tool such as Airflow or Prefect
  • Strong SQL and cloud data warehouses such as Athena, Snowflake, BigQuery or Redshift
  • Data modelling approaches such as Kimball dimensional modelling, Data Vault or one big table, and how to apply them to data platform requirements
  • Data validation and testing with tools such as Pytest
  • Git, testing frameworks, code reviews and CI/CD pipelines, applied with discipline
  • Practical use of approved AI-assisted development tools, with careful review of what they produce
  • Awareness of data privacy and POPIA requirements when handling personal and transaction data
  • dbt, including integrating dbt projects into Dagster (advantageous)
  • Containerisation and orchestration with Docker and Kubernetes (advantageous)
  • Infrastructure as code with Terraform (advantageous)
Experience & Qualifications
  • 3–5 years of professional software engineering or data engineering experience
  • Experience building, deploying and supporting production data pipelines with Dagster or a similar orchestration tool, on a cloud data warehouse
  • Relevant IT qualification or equivalent practical experience
Growth Mindset

At Shop2Shop, we believe growth is part of the job, not just for the company, but for every individual. In this role, you are expected to:

  • Embrace continuous learning and stay curious about your craft
  • Seek feedback and reflect on both successes and setbacks
  • Set personal development goals and take ownership of your growth
  • Support others in their growth journeys through knowledge sharing and collaboration
  • Stay open to change and adapt quickly in a fast-paced environment

This role is for someone who takes pride in their craft and wants to build data that people can trust without a second thought.

Other Requirements
  • Hybrid working model with regular in-office presence in Stellenbosch based on team rhythms, collaboration needs, and business requirements
  • Ability to be available for key planning sessions, stakeholder workshops, and in-person team collaboration when required
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Senior Scala Developer
Senior Scala Developer

Shop2Shop • Stellenbosch

Híbrido
ZAR 1.000.000 - 1.500.000
Data Platform Engineer - Dagster & Python
Data Platform Engineer - Dagster & Python

Shop2Shop • Stellenbosch

Híbrido
ZAR 600.000 - 1.000.000
Data Scientist
Data Scientist

Edge Executive Search Group • Centurion

Presencial
ZAR 700.000 - 1.100.000
Data Scientist
Data Scientist

Edge Executives • Centurion

Presencial
ZAR 600.000 - 900.000
Data Engineer Jhb - Northern Suburbs
Data Engineer Jhb - Northern Suburbs

Psybergate • Johannesburg

Presencial
ZAR 1.000.000 - 1.500.000
Data Engineer
Data Engineer

Communicate Recruitment • Johannesburg

Presencial
ZAR 900.000 - 1.000.000
Data Principal Engineer New Cape Town
Data Principal Engineer New Cape Town

Takealot Online Ltd. • Cape Town

Híbrido
ZAR 1.200.000 - 1.800.000
Hybrid working model
Staff discount
Birthday leave
+3
Data Principal Engineer
Data Principal Engineer

takealot.com • Cape Town

Híbrido
ZAR 1.000.000 - 1.800.000
Hybrid working model
Staff discount
Birthday leave
+2
Senior Data Engineer - Cape Town
Senior Data Engineer - Cape Town

Moyo Africa Consulting • Sudáfrica

Presencial
ZAR 900.000 - 1.500.000
Competitive salary
Health insurance
Retirement benefits
+3
Senior Data Engineer: Aws Pipelines & Ai Governance
Senior Data Engineer: Aws Pipelines & Ai Governance

Jvr Software • Sudáfrica

Híbrido
ZAR 739.000 - 1.052.000
Hybrid work arrangement
Professional development opportunities
Health and retirement benefits
+1