Data Engineer

Calibrate People; Johannesburg; Hybrid; ZAR 700,000 - 1,100,000; Full time

Job summary

Calibrate People is seeking a Data Engineer to join our Technology Team in a hybrid Johannesburg setting. The role focuses on building scalable data pipelines and ETL workflows across cloud and on-prem systems, collaborating with data scientists and analysts to deliver actionable insights.

You will leverage tools like AWS/Azure, Spark, and SQL, while ensuring data quality, security, and governance. Strong communication and problem-solving are essential for success in this fast-paced environment.

Qualifications

• Bachelor's degree from an accredited university.
• 3+ years hands-on data engineering; finance sector experience is a plus.
• Experience with on-premises and cloud data solutions.
• Familiarity with core cloud data services: Glue, Redshift, S3, Lambda, EMR, Athena, Fabric, Databricks.
• Knowledge of Spark, Hadoop, Kafka.
• Proficiency in Python and SQL.
• Experience with relational databases: RDS, MS SQL, Azure SQL DB, Postgres.
• IaC: CloudFormation, Terraform, Azure ARM/Bicep.
• CI/CD tooling experience.
• Strong communication and analytical skills.

Responsibilities

• Design and implement scalable data pipelines using Cloud services such as Glue, Redshift, S3, Lambda, EMR, Athena, Microsoft Fabric, andDatabricks.
• Develop and maintain ETL processes to transform and integrate data fromdiverse sources.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver top-notch data solutions.
• Optimize and fine-tune the performance of data pipelines and queries.
• Ensure data quality and integrity through rigorous testing andvalidation processes.
• Implement data security and compliance best practices.
• Monitor and troubleshoot data pipeline issues, ensuring timelyresolutions.
• Stay abreast of the latest developments in data engineeringtechnologies and best practices.

Skills

Data engineering

Python (Pandas)

SQL

Cloud platforms

Big data

Data modeling

CI/CD

IaC

Education

Bachelor's degree

Tools

Databricks

AWS CloudFormation

Terraform

Azure ARM/Bicep

Glue

Redshift

S3

Job description

\"Join Us as a Data Alchemist: Turn Raw Data into Golden Insights!\"

Location:Johannesburg, SA - Hybrid | Salary:Competitive (Cause you’re worth it!)

Are you a data virtuoso with a flair for crafting seamless datahighways?We're on the hunt for a Data Engineer extraordinaire to join our client’s dynamic Technology Team. This role offers the uniqueopportunity to collaborate with clients, designing secure, scalable,and cost-effective solutions that harness the power of both on-premises technologies and Cloud platforms like Azure and AWS. If youpossess a blend of Cloud expertise, hands-on data engineeringexperience, and the ability to communicate complex ideas withcreativity and clarity, we want to hear from you!

Your Mission, Should You Choose to Accept:

• Design and implement scalable data pipelines using Cloud services such asGlue, Redshift, S3, Lambda, EMR, Athena, Microsoft Fabric, andDatabricks.


• Develop and maintain ETL processes to transform and integrate data fromdiverse sources.


• Collaborate with data scientists, analysts, and other stakeholderstounderstand data requirements and deliver top-notch data solutions.


• Optimize and fine-tune the performance of data pipelines and queries.


• Ensure data quality and integrity through rigorous testing andvalidation processes.


• Implement data security and compliance best practices.


• Monitor and troubleshoot data pipeline issues, ensuring timelyresolutions.


• Stay abreast of the latest developments in data engineeringtechnologies and best practices.

Your Toolkit:

• A Bachelor's degree from an accredited university.


• 3+ years, hands-on experience in data engineering; prior experience inthe financial services industry is a plus but not mandatory. Bonuspoints if you have experience with banks and insurers!


• Proven skills in building and managing data solutions using on-premisestechnologies or Cloud platforms.


• Familiarity with core Cloud Data Services, including Glue, Redshift, S3,Lambda, EMR, Athena, Microsoft Fabric, or Databricks.


• Knowledge of big data technologies such as Apache Spark, Hadoop, orKafka.


• Proficiency in programming languages like Python (with Pandas) andSQL.


• Experience working with relational databases like AWS RDS, MS SQL,Azure SQL DB, or Postgres.


• Solid understanding of data modeling, ETL processes, and datawarehousing.


• Experience with Infrastructure as Code (IaC) tools like AWSCloudFormation, Terraform, or Azure ARM/Bicep.


• Familiarity with CI/CD tools to streamline software development anddelivery.


• Excellent communication, problem-solving, and analytical skills, withthe ability to present complex technical concepts clearly andconcisely.


• Relevant Cloud certifications are a plus, showcasing your commitmentto professional development and mastery of Cloudservices and best practices.


• Strong desire to learn and enhance business knowledge.