Data Engineer

Khonology (Pty) Ltd; Johannesburg; On-site; ZAR 600 000 - 900 000; Full time

Job summary

A digital services company in Johannesburg is seeking a skilled Data Engineer. The ideal candidate will have expertise in Python and cloud technologies (AWS/Azure). Responsibilities include designing and maintaining data pipelines, implementing ETL processes, and managing Big Data systems to support production initiatives. The role requires experience with Hadoop and Spark for efficient data processing.

Qualifications

• Strong background in Python, AWS/Azure Cloud technologies.
• Experience with data pipeline management.
• Knowledge of Big Data technologies.

Responsibilities

• Write efficient and maintainable code in Python and SQL.
• Design and maintain data pipelines for data flow.
• Implement and optimize ETL/ELT processes.

Skills

Python

AWS

Azure

SQL

ETL processes

ELT processes

Hadoop

Spark

Tools

Bitbucket

GitHub

CodeCommit

Bamboo

Octopus Deploy

CodeBuild

CodeDeploy

Job description

About the Company

Khonology is a digital services company focused on software development, Application Support, data analytics and engineering.

We are looking for a skilled Data Engineer to join our team. The ideal candidate will have a strong background in Python, AWS/Azure Cloud technologies, data pipeline management, ETL and ELT principles, and Big Data technologies. You will be responsible for designing, building, and maintaining scalable and robust data systems to support our data-driven initiatives and productionizing data related systems.

Responsibilities and Competencies

• Programming: Write efficient and maintainable code in Python and SQL.
• Data Pipeline Management: Design, develop, and maintain data pipelines to ensure smooth data flow from various sources to the data warehouse.
• Production Systems: Apply best practices in production systems management to ensure reliability and scalability.
• ETL and ELT Processes: Implement and optimize ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes to ensure data is correctly ingested, transformed, and loaded.
• Data API Integration: Develop and maintain data APIs to facilitate seamless data access and integration with other systems.
• Big Data Technologies: Utilize Hadoop and Spark to process large volumes of data efficiently.
• Database Management: Manage and optimize relational databases to ensure high performance and reliability.
• Data Modeling: Design and implement data models to support data product and machine learning needs.
• Query Optimization: Ensure that queries are always optimized for performance to enhance system efficiency.

Our current tech stack

The ideal candidate should be familiar with our current tech stack:

• Platforms: AWS, Microsoft Azure
• Main Programming language: Python, SQL (Domain-Specific Language)
• Big Data technologies (On prem): Hadoop, Spark
• Code Repositories: Bitbucket, GitHub, CodeCommit
• Deployment frameworks: Bamboo, Octopus Deploy, CodeBuild and CodeDeploy