Senior Data Engineer, Unified Platform

Drweng; Chicago, New York (IL, NY); On-site; USD 150,000 - 190,000; Full time

Job summary

DRW is hiring a Senior Data Engineer for the Unified Platform organization in Chicago. You will design, build, and operate the data platform and managed services used by Traders, Researchers, and Back-Office staff.

You will own engineering projects, work with platform users to solve technical problems, and contribute to metadata governance and data quality initiatives. You will mentor engineers, lead design reviews, and drive automation, observability, and scalable data processing across batch

Qualifications

• 7+ years of experience with modern data technologies or distributed data systems.
• Strong Java and/or Python production experience, plus SQL, Bash and Linux familiarity.
• Experience building/operating managed data or infrastructure services and production support.
• Ownership of engineering projects from design through deployment and operation.
• Understanding of distributed systems trade-offs like fault tolerance and concurrency.
• Experience with batch/streaming processing: state management, retries, replay, data consistency.
• Experience with data processing/query tech (Flink, Spark, ClickHouse) and various storage systems.

Responsibilities

• Design, build, and operate managed platform services for data storage and processing.
• Own designs and implementations for platform services and lead delivery from requirements to deployment.
• Automate provisioning, configuration, upgrades, and monitoring of managed services.
• Build and integrate metadata management tools, data catalogs, and lineage tracking.
• Implement schema governance, data quality checks, and access controls with security stakeholders.
• Diagnose integration and performance issues with platform users and improve services.
• Improve reliability via observability, capacity management, and testing.
• Participate in production support and incident response for platform services.
• Deliver platform upgrades and data migrations with validation and compatibility checks.
• Lead design and code reviews and mentor junior engineers.

Skills

Java

Python

SQL

Bash

Linux

Data platforms

Ownership

Tools

Flink

Spark

ClickHouse

Iceberg

Delta Lake

Kafka

Parquet

Protobuf

Avro

Job description

DRW is a diversified trading firm with over 3 decades of experience bringing sophisticated technology and exceptional people together to operate in markets around the world. We value autonomy and the ability to quickly pivot to capture opportunities, so we operate using our own capital and trading at our own risk. Headquartered in Chicago with offices throughout the U.S., Canada, Europe, and Asia, we trade a variety of asset classes including Fixed Income, ETFs, Equities, FX, Commodities and Energy across all major global markets. We have also leveraged our expertise and technology to expand into three non-traditional strategies: real estate, venture capital and cryptoassets. We operate with respect, curiosity and open minds. The people who thrive here share our belief that it's not just what we do that matters- it's how we do it. DRW is a place of high expectations, integrity, innovation and a willingness to challenge consensus.

Role

As a Senior Data Engineer in our Unified Platform organization, you will design, build, and operate the data platform and managed services used by Traders, Quantitative Researchers, and Back-Office personnel. You will own engineering projects and service components, work directly with platform users to solve technical problems, and contribute to metadata management and governance capabilities. This is a hands-on role combining technical design ownership, software development, production support, and mentorship within the broader platform architecture.

Key Responsibilities

• Design, build, and operate managed platform services for data storage, distributed query execution, batch and streaming processing, and workload orchestration.
• Own technical designs and implementation decisions for services and components within the broader platform architecture, and lead their delivery from requirements through deployment and support.
• Automate provisioning, configuration, upgrades, and monitoring for managed services, reducing manual operations and improving consistency across deployments.
• Build and integrate metadata management tools, including data catalogs and lineage tracking, to help users discover datasets and understand their structure, ownership, and dependencies.
• Implement schema governance, data quality checks, access controls, and usage tracking in line with established policies, working with security and data stakeholders on technical requirements.
• Work with users of managed services to diagnose integration and performance issues, clarify technical requirements, and improve the services their workloads depend on.
• Improve the reliability and performance of managed services and shared processing infrastructure through observability, capacity management, and testing.
• Participate in production support and incident response for these services, resolving root causes and maintaining operational runbooks.
• Deliver platform upgrades and data migrations with appropriate validation and compatibility checks to protect downstream workloads.
• Lead design and code reviews for the services you own, document technical trade-offs, and contribute improvements to architectural patterns shared across the team.
• Mentor less experienced engineers through hands-on collaboration and feedback, promoting maintainable code, automated testing, and sound operational practices.

Required Qualifications

• 7+ years of experience working with modern data technologies and/or building data-intensive distributed systems.
• Expertise in Java and/or Python, with a proven ability to deliver high-quality, maintainable production software, and strong familiarity with SQL, Bash, and Linux environments.
• Experience building and operating managed data or infrastructure services offered to other teams, including operational automation and production support.
• Demonstrated ownership of engineering projects or service components from technical design through deployment and ongoing operation.
• Strong understanding of distributed systems trade-offs, including fault tolerance, concurrency, partitioning, and recovery from partial failures.
• Production experience with batch and streaming systems, including practical knowledge of state management, retries, replay, and data consistency.
• Experience with data processing or query technologies such as Flink, Spark, or ClickHouse , and the ability to evaluate their performance and operational trade-offs.
• Experience with multiple storage technologies, including object storage, relational or NoSQL databases, and event streaming systems such as Kafka.
• Familiarity with open table formats such as Iceberg or Delta Lake.
• Practical experience with data modeling, schema governance and evolution, and data quality controls.
• Experience integrating metadata management, data cataloging, or lineage capabilities into data systems.
• Experience with data formats and serialization systems such as Arrow, Parquet, Protobuf , or Avro.
• Experience with cloud-native data infrastructure, including Kubernetes, workflow orchestration, automated deployment, and production monitoring.
• Strong communication and problem-solving skills, with the ability to troubleshoot issues with platform users, collaborate across teams, prioritize competing work, and deliver in a fast-paced trading environment.

Desirable Qualifications

• Experience extending or scaling data catalogs or lineage tracking tools across multiple storage and compute technologies.
• Experience implementing access controls, usage tracking, or other governance capabilities in shared data environments.
• Familiarity with financial datasets and the needs of trading, quantitative research, or back-office users.

The annual base salary range for this position is $160,000 to $25