Senior Data Engineer

Liquid Thought; Cape Town; On-site; ZAR 800,000 - 1,200,000; Full time

Job summary

A leading company is seeking a Senior Data Engineer to build a scalable technology platform aimed at helping young job seekers. The role focuses on managing large datasets, implementing solutions on a cloud-native stack primarily using Google Cloud, Python, and MySQL. Ideal candidates will have extensive experience in data engineering and a background in science or engineering.

Qualifications

• 8-12 years of experience as a Data Engineer.
• Experience with real-time and batch data processing.
• Understanding of data models and relationships.

Responsibilities

• Collect, store, and process large datasets.
• Design data solutions for real-time and batch processing.
• Collaborate with stakeholders to align on data objectives.

Skills

Data processing

Python

MySQL

ETL processes

Cloud platforms

Statistical analysis

Education

Degree or Diploma in Science or Engineering

Tools

Google Cloud Platform

Apache Beam

Apache Spark

Job description

Location : Cape Town / Gauteng (Hybrid / Remote, once a week in office)

About the Role :

Join us in building a groundbreaking, scalable, future-proof technology platform that transforms lives by helping young, first-time job seekers find employment and work experiences. As a technical specialist, you will assist colleagues in creating, maintaining, and utilizing the data stack.

Objective :

• The Senior Data Engineer's goal is to collect, store, and process large datasets, providing optimal solutions for their use, along with implementation, improvement, and monitoring.
• The technology stack is primarily cloud-native, hosted on Google Cloud Platform, with services developed in .Net Core and Python.
• You will partner with the Head of Data on database design and strategy to support our client's future plans and technology roadmap.
• Leverage Big Data to help clients find their first job, implement an Agile Development Program within a DevOps culture, and enable data-driven decision-making.
• The data landscape includes MySQL, BigQuery, and Composer; this stack is currently being revised to simplify, empower business users, and enhance data integrity.

Minimum Requirements :

• Degree or Diploma in Science or Engineering (Mathematics, Statistics, Computer Science, or related fields)
• 8-12 years of experience as a Data Engineer
• Proficiency with data tools and programming languages
• Experience with real-time and batch data processing
• Understanding of data models and relationships
• Experience with large-scale ETL processes (Apache Beam, Apache Spark, SSIS)
• Cloud platform experience (Google Cloud, Azure, or AWS)
• Proficiency in Python and MySQL
• R knowledge is a plus

Job Description :

You will serve as a technical specialist, supporting colleagues in the development, maintenance, and utilization of the data stack.

Planning and Strategy :

• Collaborate with stakeholders to understand business and data objectives aligned with our client's strategy.
• Design data solutions for both real-time and batch processing environments.
• Document the planning, implementation, and operation of the data platform.
• Select and integrate big data tools and frameworks to meet business and analysis needs.
• Coordinate with data analysts, testers, developers, and business analysts to deliver data requirements.

Execution :

• Advise on data platform usage, including analysis, security, architecture, and modeling.
• Ensure infrastructure supports large transactional data volumes.
• Monitor performance and recommend infrastructure improvements.
• Collaborate with platform developers for seamless data ingestion.
• Build data structures and pipelines for storage and transformation.
• Work with data analysts to ensure accurate, accessible data for downstream use.

Support :

• Assist teams with ad hoc data analysis for business insights.
• Support data validation and quality assurance for dashboards and reports.