Senior Data Engineer

Harambee Youth Employment Accelerator; Johannesburg; On-site; ZAR 400 000 - 800 000; Full time

Job summary

Join a dynamic team at a forward-thinking organization that is dedicated to transforming lives through technology. As a Senior Data Engineer, you will play a crucial role in designing and implementing data solutions that empower young job seekers. This innovative firm is building a scalable technology platform that utilizes big data to facilitate personalized employment journeys. You'll collaborate closely with various stakeholders, ensuring that the data infrastructure is robust and efficient. If you are passionate about making a difference and thrive in a collaborative environment, this opportunity is perfect for you.

Qualifications

• 8-12 years of experience as a Data Engineer is required.
• Strong SQL and Python skills are essential for this role.
• Experience with large scale ETL tools is necessary.

Responsibilities

• Design data solutions for real-time and batch environments.
• Implement data retention policies and monitor performance.
• Support teams with data analysis and ensure data quality.

Skills

SQL proficiency

Python

Data tools and languages

Realtime and batch processing

Large scale ETL (Apache beam or Apache spark, SSIS)

Document stores

Cloud experience (Google Cloud Platform, Azure, or AWS)

Education

Degree or diploma in Science or Engineering

Tools

Google Cloud Platform

MySQL

Job description

ABOUT THE ROLE

The Harambee Software Development team is looking for an exceptional Senior Data Engineer to join their dynamic team based in Johannesburg. The Senior Data Engineering role's objective is collecting, storing and processing large sets of data, with the primary goal of providing optimal solutions to use and of course the implementation, improvement, and monitoring thereof. The Senior Data Engineer is also responsible to partner with the Head of Data on database design and strategy to enable Harambee’s future strategy and technology roadmap. Furthermore, the Data Engineer will be doing all that in an organisation that changes lives and has a culture of teamwork, openness and collaboration. If this prospect excites you, then you may be the person we are looking for.

The Technical Challenge Harambee is building a ground-breaking, scalable, future-fit technology platform that changes lives by helping young first-time work-seekers find jobs and work experiences. Our technology needs to cater for the unique, personalised employment journey of each workseeker – ultimately using the power of big data to help them find their first job. We run an Agile Development Programme in a DevOps infrastructure and culture. Our cloud-native technology stack is predominantly hosted in Google Cloud Platform with services written in .Net Core and Python. The data landscape includes MYSQL, Big Query and Composer. This technology stack is currently being revised to drive a strategy of simplification, business user enablement and data integrity. Google did a case study on us.

READ IT (https://cloud.google.com/customers/harambee/)

JOB DESCRIPTION

You will work as a technical specialist, aiding your colleagues in the creation, maintenance and use of the data stack.

1. Planning and Strategy:
2. Engage with various stakeholders to understand the business and data goals in context of Harambee’s strategy and vision.
3. Design data solutions for real-time and batch environments.
4. Document the planning, implementation and operation of the data platform.
5. Select and integrate any big data tools and frameworks required to enable the business and data analysis goals.
6. Work closely with the data analyst, testers, developers and business analysts to design and ensure delivery of data requirements.
7. Execution:
8. Assist with and advise on the data platform and its use, including data analysis, security, architecture and modelling.
9. Ensure effective infrastructure for processing large volumes of transactional data.
10. Implement data retention policies.
11. Monitor performance and advise on necessary infrastructure changes.
12. Work with platform developers to ensure seamless and optimal data ingestion.
13. Build data structures and pipelines that ensure the storage and transformation of data.
14. Work with the data analysts to ensure necessary and accurate data is available for downstream use.
15. Support:
16. Support other teams with ad hoc data analysis to derive business insights.
17. Support the Data analysis and Business Intelligence team on data validation and quality assurance to ensure accurate dashboards and reports.

MINIMUM REQUIREMENTS

1. Degree or diploma in Science or Engineering (Mathematics, Statistics, Computer Science or related field).
2. Minimum 8 - 12 years’ experience as a Data Engineer.
3. Knowledge of and use with data tools and languages.
4. SQL proficiency.
5. Document stores.
6. Realtime and batch processing.
7. Models and relationships.
8. Large scale ETL (Apache beam or Apache spark, SSIS).
9. Cloud experience (Google Cloud Platform, Azure, or AWS).
10. Python.
11. MySQL.
12. R (nice to have).
13. Has worked at a company of similar size and complexity to Harambee over the past 24 months.
14. Importantly, you must fit in with the Harambee culture, be passionate about our mission and be an individual who leaves things better than you found them.
15. Keen to work for purpose and be inspired by the bigger social problem we want to solve. Keen to learn and grow.

COMPETENCIES

1. Curiosity and drive to understand, improve and deliver.
2. Proven track record of quality delivery.
3. Comfortable explaining data models, data structures and analytical findings to technical and non-technical audiences.
4. Willingness to learn.
5. A passion for knowledge sharing and upskilling other team members.
6. Highly detail oriented.