Senior Data Engineer: Scalable Batch & Streaming on Hybrid Cloud

Jobot; Chicago (IL); On-site; USD 180,000 - 250,000; Full time

Job summary

Jobot in Chicago seeks a Lead Data Engineer to architect scalable batch and streaming pipelines in a hybrid cloud environment.

You will own the data platform design, mentor engineers, and collaborate with Trading, Quant, Technology, and Business Operations to deliver high-impact data solutions that drive growth.

Requirements include 5+ years of software/data engineering, Java/Scala, AWS, Spark, Kubernetes, and Databricks experience.

Qualifications

• BS/MS/PhD in Computer Science, Engineering, Physics, Math, or equivalent field.
• 5+ years of professional experience developing software applications.
• Java/Scala required; Python experience is a plus.
• Hands-on with data platforms like Delta Lake, Spark, Kubernetes, Kafka, and Databricks.
• Experience building large-scale batch and streaming pipelines with strict SLA and data quality.
• Excellent communication, analytical, and problem-solving skills.
• Experience with AWS Cloud development, deployment, and monitoring.

Responsibilities

• Design and expand the data platform across multiple data sources.
• Build and deploy batch and streaming pipelines using Kubernetes, Kafka, and Databricks.
• Collaborate with Trading, Quant, Technology, and Operations teams.
• Mentor junior engineers on software and data engineering best practices.
• Ensure data quality, SLA, and observability through automated tests.

Skills

Java/Scala

Python

AWS Cloud

Spark

Kubernetes

Kafka

Databricks

Delta Lake

ClickHouse

Presto/Trino

Big Data

SQL

Education

BS/MS/PhD in Computer Science/Engineering/Physics/Math, or equivalent

Tools

Databricks

Kubernetes

Kafka

Spark

Delta Lake

Job description

Jobot in Chicago seeks a Lead Data Engineer to architect scalable batch and streaming pipelines in a hybrid cloud environment.

You will own the data platform design, mentor engineers, and collaborate with Trading, Quant, Technology, and Business Operations to deliver high-impact data solutions that drive growth.

Requirements include 5+ years of software/data engineering, Java/Scala, AWS, Spark, Kubernetes, and Databricks experience.