Senior Data Engineer: Real-Time Data Platforms

Selby Jennings; Chicago (IL); On-site; USD 110,000 - 150,000; Full time

Job summary

Selby Jennings in Chicago is seeking an experienced Data Engineer to design and support scalable data platforms powering analytics and critical business operations. The role requires a hands-on contributor who thrives in a fast-paced, data-intensive environment.

Responsibilities include building large-scale data infrastructure, real-time and batch processing with Kafka, Spark, and cloud stores (S3/HDFS), and ensuring data reliability.

Qualifications

• 5+ years of experience designing and supporting data engineering solutions in large-scale production environments.
• Experience building and supporting streaming data platforms (e.g., Kafka).
• Hands-on experience with data lake/cloud storage/distributed platforms (S3, HDFS, Databricks, Iceberg).
• Strong experience with Spark, Flink, or similar distributed processing technologies.
• Proficiency in Java, Python, and SQL.
• Experience using Python-based analytics, data science, or ML libraries.
• Knowledge of Docker and Kubernetes.
• Experience with observability, monitoring, and alerting in distributed environments.
• Strong problem-solving and debugging in Linux/Unix; scripting with Python/Bash.

Responsibilities

• Design, implement, and maintain large-scale data infrastructure and distributed data platforms.
• Develop and support real-time and batch data processing systems using streaming and big data technologies.
• Build robust data pipelines to ingest, transform, and deliver data across the organization.
• Collaborate with software engineers, platform teams, and stakeholders to define data models and optimize data flows.
• Ensure the reliability, quality, and availability of enterprise data assets.
• Provide guidance on data architecture and engineering best practices.
• Monitor and troubleshoot data systems, driving root cause analysis and continuous improvement.

Skills

Kafka

Databricks

Spark

Java

Python

SQL

Docker

Kubernetes

AWS

Linux

Bash

Data modeling

Job description

Selby Jennings in Chicago is seeking an experienced Data Engineer to design and support scalable data platforms powering analytics and critical business operations. The role requires a hands-on contributor who thrives in a fast-paced, data-intensive environment.

Responsibilities include building large-scale data infrastructure, real-time and batch processing with Kafka, Spark, and cloud stores (S3/HDFS), and ensuring data reliability.