Real-Time Data Engineer: ML Pipelines & APIs

Boardroom Appointments; Gqeberha; On-site; ZAR 550,000 - 900,000; Full time

Job summary

Boardroom Appointments is seeking an Intermediate Data Engineer to design and scale data infrastructure that supports real-time processing of massive datasets. You will develop scalable ingestion pipelines and ML inference workflows, and construct APIs to deliver analytics outputs to multiple business units.

The role requires strong programming in Scala or Java, solid data modeling skills, and experience with cloud platforms (AWS/GCP/Azure).

Qualifications

• Matric (Grade 12) required.
• Masters in Software Engineering, Data Engineering, Computer Science or related field.
• 2-3 years of relevant work experience.
• Strong Scala or Java background.
• Knowledge of AWS, GCP, Azure, or other cloud platform.
• Understanding of data modeling principles.
• Ability to work with complex data models.
• Experience with relational and NoSQL databases (Postgres, ElasticSearch/OpenSearch, graph databases such as Neptune or neo4j).
• Experience with Spark, Hadoop, Kafka, Docker, Kubernetes or other distributed computing systems.
• Knowledge of API development and machine learning deployment.

Responsibilities

• Build and scale data infrastructure that powers real-time data processing of billions of records in a streaming architecture.
• Build scalable data ingestion and machine learning inference pipelines.
• Build general-purpose APIs to deliver data science outputs to multiple business units.
• Scale up production systems to handle increased demand from new products, features, and users.
• Provide visibility into the health of our data platform and optimize cloud costs.
• Automate and handle the life-cycle of the systems and platforms that process our data.

Skills

2-3 years experience

Scala or Java

Cloud platforms

Data modeling

Complex data models

API development

ML deployment

Education

Matric (Grade 12)

Masters in Software Engineering, Data Engineering, Computer Science or related field

Tools

Spark

Hadoop

Kafka

Docker

Kubernetes

Postgres

ElasticSearch/OpenSearch

Neo4j/Neptune

Job description

Boardroom Appointments is seeking an Intermediate Data Engineer to design and scale data infrastructure that supports real-time processing of massive datasets. You will develop scalable ingestion pipelines and ML inference workflows, and construct APIs to deliver analytics outputs to multiple business units.

The role requires strong programming in Scala or Java, solid data modeling skills, and experience with cloud platforms (AWS/GCP/Azure).