Data Platform SRE: Kubernetes, Kafka & Big Data

IMC; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

IMC is seeking a Data Platform Engineer to safely receive and store trading-related data and to operate and improve shared data access systems. You will work on distributed storage, data pipelines, and heavy automation within a global trading environment based in Sydney.

Ideal candidates bring Linux administration experience, real-world exposure to financial markets, and hands-on work with Docker, Kubernetes, Kafka, Spark, HDFS, and orchestration tools such as Airflow or Prefect.

Qualifications

• 2+ years of experience managing large-scale multi-petabyte data infrastructure in a similar role.
• Knowledge of Linux system administration and internals with ability to troubleshoot issues in Linux environments.
• Experience with at least one of Kafka, Spark, Cassandra/Scylla, or HDFS.
• Working knowledge of Docker, Kubernetes, and Helm.
• Experience with data access technologies such as Dremio and Presto.
• Familiarity with workflow orchestration tools like Airflow and Prefect.
• Exposure to cloud platforms such as AWS, GCP, or Azure.

Responsibilities

• Manage and monitor distributed systems, storage infrastructure, and data processing platforms (HDFS, Kubernetes, Dremio, in-house pipelines).
• Drive automation and CI/CD to enable rapid deployment of hardware and software solutions.
• Collaborate with systems and network engineers, traders, and developers to support and troubleshoot queries.
• Stay up to date with emerging technologies; evaluate and implement innovative solutions.

Skills

Linux administration

Systems automation

Cross-functional collaboration

Troubleshooting

Tools

Kafka

Spark

Cassandra/Scylla

HDFS

Docker

Kubernetes

Helm

Airflow

Prefect

Dremio

Presto

Job description

IMC is seeking a Data Platform Engineer to safely receive and store trading-related data and to operate and improve shared data access systems. You will work on distributed storage, data pipelines, and heavy automation within a global trading environment based in Sydney.

Ideal candidates bring Linux administration experience, real-world exposure to financial markets, and hands-on work with Docker, Kubernetes, Kafka, Spark, HDFS, and orchestration tools such as Airflow or Prefect.