Senior Data Engineer: Scalable Batch & Streaming Pipelines

Jobot; Chicago (IL); On-site; USD 180,000 - 250,000; Full time

Job summary

Jobot is seeking a Lead Data Engineer in Chicago, IL to own design and expansion of our data platform, driving batch and streaming pipelines across hybrid cloud using Kubernetes, Kafka, and Databricks. You will mentor engineers, deliver high-quality code, and ensure data quality and SLAs in a fast-paced, Agile environment.

Requirements include 5+ years in software/data engineering, Java/Scala, Python, and hands-on AWS cloud experience; familiarity with Delta Lake, Spark, Kubernetes, and data

Qualifications

• BS/MS/PhD in Computer Science or equivalent technical field.
• 5+ years of professional software/data engineering experience.
• Proficiency in Java/Scala; Python is a plus.
• Experience with data platforms and technologies like Delta Lake, Spark, Kubernetes, Kafka, Databricks.
• Experience building large-scale batch and streaming pipelines with SLA and data quality requirements.
• Strong communication, analytical and problem-solving skills.

Responsibilities

• Build and deploy batch and streaming pipelines across a hybrid cloud architecture using Kubernetes/EKS, Kafka/MSK, and Databricks/Spark.
• Mentor junior engineers in software and data engineering best practices.
• Produce clean, well-tested, documented code for mission-critical applications.
• Create automated data validation test suites to enforce data quality and SLA commitments.
• Collaborate with Trading, Quant, Technology, and Business Operations to define data production/consumption workflows.

Skills

Java/Scala

Python

Data engineering

Batch & streaming

Education

BS/MS/PhD in CS/Engineering/Math

Tools

Kubernetes

Kafka/MSK

Databricks

Spark

Delta Lake

ClickHouse

Presto/Trino

AWS Cloud

Job description

Jobot is seeking a Lead Data Engineer in Chicago, IL to own design and expansion of our data platform, driving batch and streaming pipelines across hybrid cloud using Kubernetes, Kafka, and Databricks. You will mentor engineers, deliver high-quality code, and ensure data quality and SLAs in a fast-paced, Agile environment.

Requirements include 5+ years in software/data engineering, Java/Scala, Python, and hands-on AWS cloud experience; familiarity with Delta Lake, Spark, Kubernetes, and data