Data Engineer

Jobot; Chicago (IL); On-site; USD 180,000 - 250,000; Full time

Job summary

Jobot in Chicago seeks a Lead Data Engineer to architect scalable batch and streaming pipelines in a hybrid cloud environment.

You will own the data platform design, mentor engineers, and collaborate with Trading, Quant, Technology, and Business Operations to deliver high-impact data solutions that drive growth.

Requirements include 5+ years of software/data engineering, Java/Scala, AWS, Spark, Kubernetes, and Databricks experience.

Qualifications

• BS/MS/PhD in Computer Science, Engineering, Physics, Math, or equivalent field.
• 5+ years of professional experience developing software applications.
• Java/Scala required; Python experience is a plus.
• Hands-on with data platforms like Delta Lake, Spark, Kubernetes, Kafka, and Databricks.
• Experience building large-scale batch and streaming pipelines with strict SLA and data quality.
• Excellent communication, analytical, and problem-solving skills.
• Experience with AWS Cloud development, deployment, and monitoring.

Responsibilities

• Design and expand the data platform across multiple data sources.
• Build and deploy batch and streaming pipelines using Kubernetes, Kafka, and Databricks.
• Collaborate with Trading, Quant, Technology, and Operations teams.
• Mentor junior engineers on software and data engineering best practices.
• Ensure data quality, SLA, and observability through automated tests.

Skills

Java/Scala

Python

AWS Cloud

Spark

Kubernetes

Kafka

Databricks

Delta Lake

ClickHouse

Presto/Trino

Big Data

SQL

Education

BS/MS/PhD in Computer Science/Engineering/Physics/Math, or equivalent

Tools

Databricks

Kubernetes

Kafka

Spark

Delta Lake

Job description

Job details:

Lead Data Engineer | 5+ Years Experience | Scalable Batch & Streaming Pipelines | Cloud Architecture (AWS, Databricks, Kubernetes)

Salary: $180,000 - $250,000 per year

About us

We are a fast-growing, tech-driven firm operating in a fast-paced market environment where data is central to our competitive edge. Our technology organization values high-performance engineering, hybrid-cloud scalability, and rigorous data quality. We foster a collaborative, Agile culture where engineers enjoy deep technical ownership, mentor peers, and directly impact mission-critical operation

Why join us?

Drive High-Impact Innovation: You will have full ownership of the architecture, pushing our data platform beyond current tech limits to support rapid business growth.

Job Details

In this role, you will: Work within a growing Data Engineering division supporting the strategic role of data at the firm. Drive the ongoing design and expansion of our data platform across a wide variety of data sources, supporting an array of streaming, operational, and research workflows. Work closely with Trading, Quant, Technology, and Business Operations teams throughout the firm to identify how data is produced and consumed, helping to define and deliver high-impact projects. Build and deploy batch and streaming pipelines to collect and transform our rapidly growing Big Data set within our hybrid cloud architecture utilizing Kubernetes/EKS, Kafka/MSK, and Databricks/Spark. Mentor junior engineers in software and data engineering best practices. Produce clean, well-tested, and documented code with a clear design to support mission-critical applications. Build automated data validation test suites that ensure that data is processed and published in accordance with well-defined Service Level Agreements (SLAs) pertaining to data quality, data availability, and data correctness. Challenge the status quo and help push our organization forward as we grow beyond the limits of our current tech stack. Qualities that make great candidates: BS/MS/PhD in Computer Science, Engineering, Physics, Math, or equivalent technical field.5+ years of professional experience developing software applications. Java/Scala experience required; Python experience is a significant plus. Highly motivated and willing to take ownership of high-impact projects upon arrival. Prior hands‑on experience with data platforms and technologies such as Delta Lake, Spark, Kubernetes, Kafka, Click House, and/or Presto/Trino. Experience building large-scale batch and streaming pipelines with strict SLA and data quality requirements. Must possess excellent communication, analytical, and problem‑solving skills. Demonstrated experience working with diverse data sets and frameworks across multiple domains—financial data experience is not required, but a strong plus. Recent hands‑on experience with AWS Cloud development, deployment, and monitoring necessary. Demonstrated experience working on an Agile team employing software engineering best practices, such as GitOps and CI/CD, to deliver complex software projects. The ability to react quickly and accurately to rapidly changing market conditions, including the ability to quickly and accurately respond and/or solve math and coding problems, are essential functions of the role.

Jobot is an Equal Opportunity Employer. We provide an inclusive work environment that celebrates diversity and all qualified candidates receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity, religion, national origin, age (40 and over), disability, military status, genetic information or any other basis protected by applicable federal, state, or local laws. Jobot also prohibits harassment of applicants or employees based on any of these protected categories. It is Jobot’s policy to comply with all applicable federal, state and local laws respecting consideration of unemployment status in making hiring decisions.

Sometimes Jobot is required to perform background checks with your authorization. Jobot will consider qualified candidates with criminal histories in a manner consistent with any applicable federal, state, or local law regarding criminal backgrounds, including but not limited to the Los Angeles Fair Chance Initiative for Hiring and the San Francisco Fair Chance Ordinance.

Information collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal.

By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Jobot, and/or its agents and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here: jobot.com/privacy-policy