Data Engineer

Kapital Consulting; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Kapital Consulting is seeking a data engineer to design and enhance scalable ETL/ELT pipelines and real-time data solutions for large-scale financial datasets. You will build robust data processing in cloud environments, using Kafka and RabbitMQ, while ensuring data integrity and security.

The role emphasizes individual contribution within a collaborative team, with a focus on delivering high-performance engineering solutions.

Qualifications

• Experience designing ETL processes in cloud environments.
• Finance or trading tech experience.
• Strong Python programming for data pipelines.
• Extensive SQL knowledge.
• Strong collaboration and communication skills.
• Australian/NZ work rights (Permanent Resident or Citizen).

Responsibilities

• Design, develop, and improve scalable ETL/ELT pipelines.
• Build and maintain real-time data processing using Kafka and RabbitMQ.
• Optimize schema designs, indexing, and query performance on large datasets.
• Deploy and maintain data services in cloud environments with security controls.
• Collaborate with stakeholders to translate business goals into data solutions.
• Implement monitoring and automated recovery to ensure data integrity.

Skills

Python programming

SQL

Real-time data processing

Cloud data pipelines

Problem solving

Tools

Kafka

RabbitMQ

Job description

Join a leading global Financial Services organisation on the development and enhancement of realt-time data solutions. This person will be responsible for designing scalable and reliable data solutions in a small, collaborative and high-output team.

The role focuses on individual contributions and development rather than management/leadership to develop high-performance engineering solutions.

What's a typical day look like?

• Design, develop, and improve highly scalable, fault-tolerant ETL/ELT pipelines.
• Build and maintain real-time processing solutions using Kafka and RabbitMQ to ensure real-time data availability.
• Work to optimise complex schema designs, indexing, and query performance for large-scale financial datasets.
• Deploy and maintain data services in cloud environments, implementing security controls and data governance standards required for the finance industry.
• Work with business stakeholders and senior leadership to understand business objectives and translate them into practical data solutions. Support fellow data engineers through code reviews and technical guidance.
• Implement monitoring and automated recovery systems to maintain the integrity and quality of critical financial data.

About you:

• Proven experience in data engineering, with a strong background in designing and implementing ETL processes within cloud environments.
• Experience within the Finance or Trading technology sector.
• Strong programming skills in Python, with experience in developing robust, maintainable, and scalable data processing pipelines.
• Extensive SQL knowledge and experience.
• Excellent problem-solving skills and the ability to work collaboratively.
• Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.
• Permanent Resident / Citizen of Australia/NZ with full-working rights.