Data Engineer

Penta Consulting; Gauteng; Hybrid; ZAR 600,000 - 900,000; Full time

Job summary

Penta Consulting is seeking a Modern Data Engineer with strong software engineering skills to build and deploy cloud-native data platforms.

Applicants should have Python, Spark/PySpark, Databricks, Kafka, Flink, and data lake experience, plus CI/CD, Git, Docker, Terraform, and AWS exposure. Real production project experience is required.

Qualifications

• Strong Python development and SDLC understanding.
• Proficient with Git and version control.
• Experience with CI/CD pipelines and modern DevOps practices.
• Hands-on with Spark/PySpark, Databricks, Kafka and Flink.
• Experience delivering both batch and streaming data solutions.
• Experience delivering cloud-based data platforms on AWS.

Responsibilities

• Build and deploy cloud-native data platforms using Python and Spark.
• Design data pipelines with batch and streaming processing.
• Collaborate with DevOps to implement CI/CD and monitoring.

Skills

Python

Spark/PySpark

Databricks

Kafka

Flink

Data Lake architectures

CI/CD

Git

Cloud platforms (AWS)

Docker

Tools

Docker

Terraform

Monitoring tools

Job description

This role is for a Modern Data Engineer with strong software engineering skills who can build and deploy cloud-native data platforms.

Key Requirements

Software Engineering

• Strong Python development skills (primary programming language)
• Solid understanding of software engineering principles and clean coding practices
• Experience working within the Software Development Life Cycle (SDLC)
• Proficient with Git and version control
• Experience with CI/CD pipelines and modern DevOps practices
• Ability to solve problems through code, not just SQL
• Hands-on experience with modern data engineering technologies, including:
• Spark / PySpark
• Databricks
• Kafka
• Flink
• Data Lake architectures
• Experience building both batch and streaming data solutions

Practical experience delivering solutions on at least one major cloud platform:

• Amazon Web Services (AWS)

Relevant services may include:

• Data processing and analytics tools (BigQuery, Synapse, Redshift, Databricks, Dataflow, Glue, EMR, Dataproc)
• Event streaming platforms (Pub/Sub, Event Hubs, Kinesis)
• Container platforms (GKE, AKS, EKS)

DevOps & Deployment

Experience with:

• Docker
• Infrastructure as Code (Terraform or similar)
• Monitoring and observability tools

Experience should be gained through real production projects, not certifications alone.

Additional Valuable Skills

• Data modelling (relational and dimensional)
• Data warehousing concepts
• Modern orchestration and data platforms such as Databricks, Kafka and Microsoft Fabric

Ideal Candidate

We're looking for a modern data engineer with a software engineering mindset. Candidates should have proven experience delivering cloud-based data solutions using Python, Spark, CI/CD, Git and modern data platforms. Experience limited to traditional SQL, SSIS or ETL development without modern engineering practices is unlikely to meet current client requirements.

What We Value

• Continuous learning and professional development
• Ability to contribute immediately within a modern engineering environment
• Client-facing consulting experience is advantageous