Lead Data Engineer | Systematic Data Platform

Paragon Alpha - Hedge Fund Talent Business; New York (NY); On-site; USD 180,000 - 260,000; Full time

Job summary

Paragon Alpha - Hedge Fund Talent Business seeks a Lead Data Engineer to design and build data infrastructure powering investment and research workflows at a world-class trading house.

You will own critical parts of the data stack, collaborating with quantitative researchers, investment professionals, data scientists and other engineers to deliver robust, low-latency data platforms.

Qualifications

• Senior, hands-on engineering role, own critical parts of data stack.
• Focus on building robust data infrastructure, not only ETL maintenance.
• Architectural decisions and engineering standards are expected.

Responsibilities

• Architecting and building scalable batch and real-time data pipelines.
• Designing platforms for ingesting, normalising and distributing large financial and alternative datasets.
• Building robust ETL/ELT frameworks with observability, lineage and data-quality controls.
• Developing data models and APIs for researchers and investment teams.
• Improving performance, reliability and scalability of existing data infrastructure.
• Designing distributed and event-driven data processing systems.

Skills

Batch pipelines

Real-time pipelines

ETL/ELT frameworks

Data modeling

APIs

Spark

Kafka

Airflow

Snowflake

AWS

Kubernetes

Distributed systems

Tools

Spark

Kafka

Airflow

Snowflake

AWS

Kubernetes

Job description

Paragon are seeking a Lead Data Engineer to help design and build the data infrastructure underpinning investment and research workflows at a world class trading house.

This is a senior, hands-on engineering role working with large, complex financial and alternative datasets. You’ll be responsible for building scalable data platforms and pipelines that allow investment teams to consume high-quality, reliable data with low latency.

You’ll take technical ownership of critical parts of the data engineering stack, working closely with quantitative researchers, investment professionals, data scientists and other engineering teams.

The focus is on building robust data infrastructure rather than simply maintaining ETL processes. You’ll be expected to make architectural decisions, establish engineering standards and remain close to the code.

Key responsibilities include:

• Architecting and building scalable batch and real-time data pipelines
• Designing platforms for ingesting, normalising and distributing large financial and alternative datasets
• Building robust ETL/ELT frameworks with strong observability, lineage and data-quality controls
• Developing data models and APIs that make complex datasets easily consumable by researchers and investment teams
• Improving the performance, reliability and scalability of existing data infrastructure
• Designing distributed and event-driven data processing systems

Python | SQL | Spark | Kafka | Airflow | Snowflake | AWS | Kubernetes | Distributed Systems | Data Modelling | Streaming & Batch Processing