Real-Time Data Engineer - Finance, Kafka & Cloud Pipelines

Kapital Consulting; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Kapital Consulting is seeking a data engineer to design and enhance scalable ETL/ELT pipelines and real-time data solutions for large-scale financial datasets. You will build robust data processing in cloud environments, using Kafka and RabbitMQ, while ensuring data integrity and security.

The role emphasizes individual contribution within a collaborative team, with a focus on delivering high-performance engineering solutions.

Qualifications

• Experience designing ETL processes in cloud environments.
• Finance or trading tech experience.
• Strong Python programming for data pipelines.
• Extensive SQL knowledge.
• Strong collaboration and communication skills.
• Australian/NZ work rights (Permanent Resident or Citizen).

Responsibilities

• Design, develop, and improve scalable ETL/ELT pipelines.
• Build and maintain real-time data processing using Kafka and RabbitMQ.
• Optimize schema designs, indexing, and query performance on large datasets.
• Deploy and maintain data services in cloud environments with security controls.
• Collaborate with stakeholders to translate business goals into data solutions.
• Implement monitoring and automated recovery to ensure data integrity.

Skills

Python programming

SQL

Real-time data processing

Cloud data pipelines

Problem solving

Tools

Kafka

RabbitMQ

Job description

Kapital Consulting is seeking a data engineer to design and enhance scalable ETL/ELT pipelines and real-time data solutions for large-scale financial datasets. You will build robust data processing in cloud environments, using Kafka and RabbitMQ, while ensuring data integrity and security.

The role emphasizes individual contribution within a collaborative team, with a focus on delivering high-performance engineering solutions.