Data Engineer – PySpark | Python | Cloud | CI/CD – Johannesburg (Hybrid)

e-Merge IT Recruitment; Johannesburg; Hybrid; ZAR 900,000 - 1,700,000; Full time

Job summary

e-Merge IT Recruitment seeks a Data Engineer to design and maintain scalable ETL pipelines using Python, PySpark, and SQL in a Johannesburg hybrid environment.

You will build API-based ingestion workflows, deploy data workloads on AWS/GCP/Azure, and implement CI/CD with Docker and Kubernetes to ensure production-grade reliability.

Qualifications

• 7+ years of professional experience in Data Engineering or similar role developing ETL pipelines.
• Advanced Python proficiency with production-grade data applications.
• Strong hands-on experience with PySpark and distributed data processing.
• Advanced SQL skills including querying, transformations, and relational data modelling.
• Production experience with at least one major cloud platform: AWS, GCP, or Azure.
• Hands-on experience with Docker and/or Kubernetes.
• Proven experience with CI/CD, automated testing, Git, and production deployments.
• Strong understanding of RESTful APIs and API-based data ingestion.
• Solid software engineering principles and the ability to deliver clean, maintainable code.
• Proven ability to own projects end-to-end and support production systems.

Responsibilities

• Design, develop, test, and maintain scalable ETL pipelines using Python, PySpark, and SQL.
• Build and manage API-based data ingestion workflows and real-time integrations.
• Develop and optimize production-grade data pipelines for performance, reliability, and scalability.
• Build and deploy data workloads across AWS, GCP, or Azure.
• Use Docker and Kubernetes for containerized data processing and deployment.
• Implement CI/CD workflows, automated testing, Git, and production deployments.
• Improve data quality, monitoring, observability, error handling, and incident response.
• Apply software engineering best practices, including modular design, version control, and documentation.
• Collaborate with cross-functional teams to deliver reliable, business-focused data solutions.

Skills

Python

PySpark

SQL

CI/CD

REST APIs

Distributed processing

Data pipelines

Docker

Kubernetes

Tools

Docker

Kubernetes

AWS

GCP

Azure

Job description

Data Engineer – PySpark | Python | Cloud | CI/CD – Johannesburg (Hybrid)

Are you a highly skilled Data Engineer with strong production experience in Python, SQL, and PySpark?

Join a growing technology team and help build scalable data infrastructure, robust data pipelines, and API-driven integrations. We’re looking for an engineer with strong software engineering fundamentals who can develop, deploy, and maintain reliable data solutions in production.

What You’ll Do:

• Design, develop, test, and maintain scalable ETL pipelines using Python, PySpark, and SQL.
• Build and manage API-based data ingestion workflows and real-time integrations.
• Develop and optimize production-grade data pipelines for performance, reliability, and scalability.
• Build and deploy data workloads across AWS, GCP, or Azure.
• Use Docker and Kubernetes for containerized data processing and deployment.
• Implement CI/CD workflows, automated testing, Git, and production deployments.
• Improve data quality, monitoring, observability, error handling, and incident response.
• Apply software engineering best practices, including modular design, version control, and documentation.
• Collaborate with cross-functional teams to deliver reliable, business-focused data solutions.

What You’ll Bring:

• 7+ years of professional experience in Data Engineering or a similar role developing ETL pipelines.
• Advanced Python proficiency, with demonstrable experience building and maintaining production-grade data applications.
• Strong hands-on experience with PySpark and distributed data processing.
• Advanced SQL skills, including querying, transformations, and relational data modelling.
• Production experience with at least one major cloud platform: AWS, GCP, or Azure.
• Hands-on experience with Docker and/or Kubernetes.
• Proven experience with CI/CD, automated testing, Git, and production deployments.
• Strong understanding of RESTful APIs and API-based data ingestion.
• Solid software engineering principles and the ability to deliver clean, maintainable code.
• Proven ability to own projects end-to-end and support production systems.

Bonus Points For:

• Experience with Terraform, Helm, or other infrastructure-as-code tools.
• Knowledge of event-driven architecture and stream processing.
• Familiarity with AI concepts and prompt engineering.
• Experience with data security, privacy and access controls.

Location: Johannesburg (Hybrid) Type: Permanent Salary: Negotiable based on experience Ref: NG60789

Ready to make your mark?