Senior Data Engineer: Kafka, Spark & Big Data Pipelines

IMC Trading; Illinois; On-site; USD 175,000 - 225,000; Full time

Job summary

IMC Trading is seeking a Data Engineer to join our Chicago team. You will architect, develop, and deploy a full Big Data environment and build pipelines using Java, Python, Spark, and Flink. Collaborate on data modeling and capacity planning to ensure data integrity and availability.

You will serve as a subject-matter expert across data-related questions from users and developers. The role emphasizes hands-on work with modern data technologies, cloud-like data backends, and a culture of robust

Qualifications

• 5+ years experience in mature data engineering environment.

Responsibilities

• Architect, develop and deploy our Big Data environment (Kafka, Hadoop, Dremio, etc.)
• Build, deploy, and monitor data processing pipelines (Java, Python, Spark, Flink)
• Collaborate with development teams on data modeling, ingestion, and capacity planning
• Work with users to ensure data integrity and availability
• Act as a Big Data SME and consult on data-related questions from users and developers

Skills

Kafka streaming

Big Data

Spark

Flink

Java

Python

SQL

Python tooling

Kubernetes

Docker

Monitoring

Statistics

Troubleshooting

Unix scripting

Tools

Kafka

Spark

Flink

Databricks

S3

HDFS

Iceberg

Prometheus

Grafana

Docker

Kubernetes

OpsGenie

Job description

IMC Trading is seeking a Data Engineer to join our Chicago team. You will architect, develop, and deploy a full Big Data environment and build pipelines using Java, Python, Spark, and Flink. Collaborate on data modeling and capacity planning to ensure data integrity and availability.

You will serve as a subject-matter expert across data-related questions from users and developers. The role emphasizes hands-on work with modern data technologies, cloud-like data backends, and a culture of robust