Data Engineer

ATS Client; Johannesburg; On-site; ZAR 850,000 - 1,250,000; Full time

Job summary

ATS Client in Johannesburg is seeking an experienced Data Engineer to lead the delivery of a global Datalake and build data pipelines that support commercial data propositions and business performance management. You will work with Product managers and stakeholders to enable revenue growth through data-driven decision making.

Responsibilities include designing data architectures, ETL development, and ensuring data quality across on-premise and cloud platforms (AWS, Azure, Oracle Cloud).

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer or in a similar role.
• Proficiency in Databricks, SQL, Python, Scala, and Java, Oracle.
• Experience with building Graph, Vector, NoSQL and data lakehouse architectures.
• Strong understanding of data warehousing concepts and ETL processes.
• Experience with on-premise and cloud platform architectures (AWS, Azure and Oracle Cloud).
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.
• Ability to work in a fast-paced, dynamic environment.
• Outstanding verbal and written communication skills.
• Excellent interpersonal and organisational skills.
• Strong commercial awareness in all areas.

Responsibilities

• Implement and evolve the strategic data architecture for the business.
• Manage the data delivery roadmap across the function.
• Design, develop, and maintain scalable data pipelines and ETL processes using Databricks.
• Write efficient SQL queries for data extraction and transformation.
• Develop and maintain data integration workflows using Python, Scala, and Java.
• Collaborate with data scientists, analysts, and stakeholders to understand data requirements and deliver solutions.
• Ensure data quality and integrity through testing and validation.
• Monitor and troubleshoot data pipeline performance and reliability.
• Stay up-to-date with industry trends and technologies in data engineering.

Skills

Data architecture

ETL processes

Data warehousing

Problem solving

Communication

Collaboration

Fast-paced environment

Commercial awareness

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

SQL

Python

Scala

Java

Oracle

Job description

We have a new and exciting opportunity for a Data Engineer to join our newly created Group Technology function. Working closely and supporting the Chief Information & Data Officer you will lead the delivery of a Datalake for the global business and building data pipelines that support commercial data propositions we sell to our customer and deliver data to your internal stakeholders to manage business performance. You will engage with Product managers and stakeholders both internally and externally, to focus on revenue growth and enhance decision making through the 'Power of Data'.

Key Responsibilities

• Implement, evolve the strategic data architecture for the business
• Manage the data delivery roadmap for the data across the function
• Design, develop, and maintain scalable data pipelines and ETL processes using Databricks.
• Write efficient and optimized SQL queries for data extraction and transformation.
• Develop and maintain data integration workflows using Python, Scala, and Java/React
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Ensure data quality and integrity through rigorous testing and validation.
• Monitor and troubleshoot data pipeline performance and reliability.
• Stay up-to-date with the latest industry trends and technologies in data engineering.

Skills, Knowledge and Experience

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer or in a similar role.
• Proficiency in Databricks, SQL, Python, Scala, and Java, Oracle
• Experience with building Graph, Vector, NoSQL and data lakehouse architectures
• Strong understanding of data warehousing concepts and ETL processes.
• Experience with on-premise and cloud platform architectures (AWS, Azure and Oracle Cloud)
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.
• Ability to work in a fast-paced, dynamic environment.
• Outstanding verbal and written communication skills
• Excellent interpersonal and organisational skills
• Strong commercial awareness in all areas

Equal Opportunities

As a global company, we value and respect the diversity of our workforce, aiming to empower everyone to embrace each other's differences. We are committed to creating an inclusive workplace where diversity, equity, and inclusion are integral to our company and culture. We recognize the benefits of a diverse workforce, where creativity and valuing differences enable us all to thrive and sparks innovation.