Senior/ Intermediate Data Engineer

Intelligent Debt Management; Wes-Kaap; On-site; ZAR 200 000 - 300 000; Full time

Job summary

A leading debt management company in Cape Town is looking for a Senior / Intermediate Data Engineer. This role involves designing, building, and maintaining scalable data solutions using tools like Databricks, Apache Spark, and Python. The ideal candidate will have strong communication skills and a growth mindset to thrive in a collaborative culture focused on helping people. Competitive benefits include medical aid, retirement annuity, and opportunities for further study.

Benefits

Medical aid

Retirement annuity

Further study opportunities

Half day off on birthday

Healthy Together campaigns

Qualifications

• Experience with big data tools, particularly Databricks.
• Proficient in Python with mandatory PySpark experience.
• Strong background in cluster computing and CI/CD pipelines.

Responsibilities

• Build and optimize data pipelines and workflows.
• Contribute to design and implementation of data flows.
• Collaborate with teams to deliver scalable data solutions.
• Expand knowledge of modern data engineering practices.

Skills

Communication skills

Attention to detail

End-to-end data flow design

Agile methodologies

Growth mindset

Tools

Databricks

Apache Spark

Python

PySpark

Azure cloud infrastructure

Job description

Senior / Intermediate Data Engineer (Pinelands, Cape Town)

We are looking for a motivated and technically skilled Data Engineer who is eager to learn and grow. This role involves working with modern data platforms and development practices to design, build, and maintain scalable data solutions.

Key Requirements

• Core Skills:
  • Strong communication and collaboration abilities.
  • Attention to detail with a focus on data quality.
  • Experience in end-to-end data flow design and implementation.
  • Familiarity with agile ways of working.
  • Growth Mindset. Demonstrated eagerness to learn and adapt, supported by examples of applying new skills in practice.
• Technical Expertise:
  • Proven experience with big data tools, particularly Databricks.
  • Strong background in cluster computing using Apache Spark.Experience developing and maintaining CI/CD pipelines.
  • Proficiency in Python with mandatory PySpark experience.
• Beneficial Experience:
  • Prior work as a backend developer, showcasing strong software engineering fundamentals.
  • Previous experience as a data engineer.
  • Familiarity with Azure cloud infrastructure and services.

What You’ll Do

• Build and optimize data pipelines and workflows on Databricks and Spark.
• Contribute to the design and implementation end-to-end data flows and CI/CD processes.
• Collaborate with cross-functional teams to deliver reliable, scalable, and high-quality data solutions.
• Continuously expand your knowledge of modern data engineering practices.

What’s in it for you?

Being South Africa’s largest company of its kind – IDM is continuously growing, with no signs of stopping.

One of our core values is ‘growing people’ and we take that very seriously. We have internal learning opportunities to better our people, and leaders who are experts in their fields and freely impart knowledge and are open to new ideas. As one of our people said, “you can speak to anyone here. Your rank doesn’t matter. You are listened to.”

Our culture is one of collaboration, diversity, and helpfulness. That goes for our people internally and how we deal with our clients.

At IDM, you can play a part in educating and assisting real South Africans get out of tough financial situations. You can make a difference with the work you do. Our people believe this too – one said, “I wake up each day knowing I am going to make a difference.”

Other key benefits:

• We make sure our employees are mentally and physically healthy with Healthy Together campaigns throughout the year.
• We offer various rewards and awards as tokens of recognition for hard work.
• We have a vibrant office with easy access to stores and outlets if you need.
• We offer company benefits – like a half day off on your birthday, medical aid, retirement annuity, and further study opportunities should you meet relevant requirements.
• We have a culture of growth, meaning you have access to move both upwards or across in the business if you want to.