Senior Data Engineer, Unified Platform

Drweng; Chicago (IL); On-site; USD 200,000 - 250,000; Full time

Benefits offered by this job

401(k) match

Dental insurance

Health insurance

Flexible spending accounts

Job summary

Drweng is seeking a Senior Data Engineer for our Data Experience team in Chicago, IL. The ideal candidate will design and manage data pipelines, work closely with stakeholders, and enhance data quality. You will play a crucial role in data governance and the integration of vendor datasets.

This position requires 7+ years of experience with modern technologies, along with excellent problem-solving and communication skills. The annual salary ranges from $200,000 to $250,000, alongside comprehensive benefits including health insurance and a 401(k) match.

Qualifications

• 7+ years of experience with modern data technologies and data-first products.
• Excellent communication skills, written and verbal.
• Proven ability to collaborate in a fast-paced environment.

Responsibilities

• Model, build, and manage data products on DRW’s Unified Data Platform.
• Design data governance processes for dataset discovery and usage tracking.
• Monitor data ingestion pipelines and ensure data quality.

Skills

Data pipeline design

SQL

Java or Python

Data modeling techniques

Data governance

Problem-solving skills

Agile collaboration

Data processing technologies

Data pipeline orchestration

Tools

Flink

Spark

Kubernetes

Airflow

Prefect

Job description

Senior Data Engineer

Senior Data Engineer on our Data Experience team will bring vendor datasets into our data platform, govern our centralized data pipelines, consolidate data delivery infrastructure, and collaborate with Traders, Quantitative Researchers, and Back‑Office personnel to maximise the firm’s data and platform tools.

Technical requirements summary

• Extensive experience designing and building data pipelines
• Experience working within modern batch and streaming data ecosystems
• Expertise in SQL and Java or Python
• Extensive knowledge of data modeling techniques
• Ability to own the delivery of data products and work with analysts and stakeholders to understand requirements and implement solutions
• Ability to contribute to project management and project reporting
• Experience driving nontrivial data migrations across platforms
• Ability to establish, follow, and enforce data modeling standards across a large ecosystem

What you will do in this role

• Model, build, and manage data products built atop DRW’s Unified Data Platform
• Work closely with Data Strategists to determine appropriate data sources and implement processes that onboard and manage new datasets for trading, research, and back‑office purposes
• Design data governance processes that enable discovery, cost‑sharing, usage tracking, access controls, and quality control of datasets to address the needs of DRW trading teams and strategies
• Continually monitor data ingestion pipelines and data quality to ensure stability, reliability, and quality of the data, and design monitoring and quality control software and processes
• Own the technical aspects of vendor ingestion pipelines, coordinate with vendor relationship managers on upcoming changes, perform routine data operations without breaking internal users, and contribute to the team’s on‑call rotation to respond to unanticipated changes
• Rapidly respond to user requests, identify platform gaps, and self‑service opportunities that make the user experience more efficient

What you will need in this role

• 7+ years of experience working with modern data technologies and building data‑first products
• Excellent written and verbal communication skills
• Proven ability to work in a collaborative, agile, and fast‑paced environment, prioritizing multiple tasks and projects, and efficiently handling the demands of a trading environment
• Proven ability to deliver rapid results within processes that span multiple stakeholders
• Strong technical problem‑solving skills
• Extensive familiarity with SQL and Java or Python, and a proven ability to develop and deliver maintainable data transformations for production data pipelines
• Experience leveraging data modeling techniques and ability to articulate the trade‑offs of different approaches
• Experience with one or more data processing technologies (e.g. Flink, Spark, Polars, Dask)
• Experience with multiple data storage technologies (e.g. S3, RDBMS, NoSQL, Delta/Iceberg, Cassandra, ClickHouse, Kafka) and knowledge of their associated trade‑offs
• Experience with multiple data formats and serialization systems (e.g. Arrow, Parquet, Protobuf/gRPC, Avro, Thrift, JSON)
• Experience managing data pipeline orchestration systems (e.g. Kubernetes, Argo Workflows, Airflow, Prefect, Dagster) and delving deeply into low‑level details
• Proven experience in managing the operational aspects of large data pipelines such as backfilling datasets, rerunning batch jobs, monitoring event throughput, and handling dead‑letter queues
• Prior experience triaging data quality control processes, correcting data gaps and inaccuracies

The annual base salary range for this position is $200,000 to $250,000 depending on the candidate’s experience, qualifications, and relevant skill set. The position is also eligible for an annual discretionary bonus. DRW offers a comprehensive suite of employee benefits including group medical, pharmacy, dental and vision insurance, 401(k) (with discretionary employer match), short and long‑term disability, life and AD&D insurance, health savings accounts, and flexible spending accounts.

For more information about DRW’s processing activities and our use of job applicants’ data, please view our Privacy Notice at https://drw.com/privacy-notice.

California residents, please review the California Privacy Notice for information about certain legal rights at https://drw.com/california-privacy-notice.