Senior Data Engineer

Selby Jennings; Chicago (IL); On-site; USD 110,000 - 150,000; Full time

Job summary

Selby Jennings in Chicago is seeking an experienced Data Engineer to design and support scalable data platforms powering analytics and critical business operations. The role requires a hands-on contributor who thrives in a fast-paced, data-intensive environment.

Responsibilities include building large-scale data infrastructure, real-time and batch processing with Kafka, Spark, and cloud stores (S3/HDFS), and ensuring data reliability.

Qualifications

• 5+ years of experience designing and supporting data engineering solutions in large-scale production environments.
• Experience building and supporting streaming data platforms (e.g., Kafka).
• Hands-on experience with data lake/cloud storage/distributed platforms (S3, HDFS, Databricks, Iceberg).
• Strong experience with Spark, Flink, or similar distributed processing technologies.
• Proficiency in Java, Python, and SQL.
• Experience using Python-based analytics, data science, or ML libraries.
• Knowledge of Docker and Kubernetes.
• Experience with observability, monitoring, and alerting in distributed environments.
• Strong problem-solving and debugging in Linux/Unix; scripting with Python/Bash.

Responsibilities

• Design, implement, and maintain large-scale data infrastructure and distributed data platforms.
• Develop and support real-time and batch data processing systems using streaming and big data technologies.
• Build robust data pipelines to ingest, transform, and deliver data across the organization.
• Collaborate with software engineers, platform teams, and stakeholders to define data models and optimize data flows.
• Ensure the reliability, quality, and availability of enterprise data assets.
• Provide guidance on data architecture and engineering best practices.
• Monitor and troubleshoot data systems, driving root cause analysis and continuous improvement.

Skills

Kafka

Databricks

Spark

Java

Python

SQL

Docker

Kubernetes

AWS

Linux

Bash

Data modeling

Job description

We are looking for an experienced Data Engineer to join a high-performing technology team in Chicago. This individual will play a key role in designing and supporting scalable data platforms that power critical business operations and analytics. The ideal candidate thrives in a fast-paced, data-intensive environment and enjoys solving complex engineering challenges at scale.

Responsibilities

• Design, implement, and maintain large-scale data infrastructure and distributed data platforms.
• Develop and support real-time and batch data processing systems using modern streaming and big data technologies.
• Build robust data pipelines and workflows to efficiently ingest, transform, and deliver data across the organization.
• Partner with software engineers, platform teams, and business stakeholders to define data models, optimize data flows, and support long-term capacity planning.
• Ensure the reliability, quality, and availability of enterprise data assets.
• Provide technical guidance and subject matter expertise on data architecture, data engineering best practices, and platform capabilities.
• Monitor and troubleshoot data systems, driving root cause analysis and continuous improvement efforts.

Qualifications

• 5+ years of experience designing and supporting data engineering solutions in large-scale production environments.
• Experience building and supporting streaming data platforms, including technologies such as Kafka or similar messaging systems.
• Hands-on experience working with modern data lake, cloud storage, or distributed data platforms such as S3, HDFS, Databricks, Iceberg, or related technologies.
• Strong experience developing data applications using frameworks such as Spark, Flink, or comparable distributed processing technologies.
• Proficiency in Java, Python, and SQL.
• Experience leveraging Python-based analytics, data science, or machine learning libraries.
• Strong understanding of containerization and orchestration technologies, including Docker and Kubernetes.
• Experience with observability, monitoring, and alerting tools in distributed environments.
• Strong analytical and problem-solving skills, with the ability to investigate and resolve complex production issues.
• Experience working in Linux/Unix environments and developing automation through scripting languages such as Python and Bash.

Preferred Background

• Experience supporting real-time, high-throughput data ecosystems.
• Familiarity with data governance, data quality, and operational excellence practices.
• Ability to collaborate effectively across engineering, platform, and business teams in a highly technical environment.