Data Platform Engineering Lead

Hedge Fund; New York (NY); On-site; USD 150,000 - 200,000; Full time

Job summary

A leading Hedge Fund in New York is seeking a highly experienced Data Platform Engineering Lead to own and scale its data platform. This hands-on role requires leadership in architecture and data governance, ensuring reliable data delivery for investment decisions.

The ideal candidate will have over 10 years of data engineering experience, particularly with Databricks and AWS. Responsibilities include building a high-performing team, maintaining data standards, and enabling AI capabilities across the organization.

Qualifications

• 10+ years of hands-on data engineering experience.
• Proven success with Databricks and AWS data platforms.
• Strong data governance background.

Responsibilities

• Own and operate the enterprise data platform end to end.
• Set standards for data modeling and governance.
• Build and mentor a high-performing data team.

Skills

Hands-on data engineering

Data modeling

Databricks

AWS

Python

SQL

Data governance

Tools

Databricks

AWS

Airflow

dbt

Job description

Multi-manager hedge fund managing over $3 billion in AUM is seeking a highly experienced technical Data Platform Engineering Lead to own and scale the firm’s data platform. This is a hands‑on architecture and leadership role responsible for delivering reliable, well‑governed data to the Front Office, Risk, and Business teams, supporting better investment decisions and stronger returns.

The role balances building the platform with making data usable for the people and systems that rely on it. The priority is to advance the architecture, raise the bar on data modeling and governance, and expand coverage and AI‑enablement in deliberate phases. It suits an engineer who leads from the front, works closely with technology teams and the business to build what users actually need, and sets clear data standards.

Key Responsibilities

• Own and operate the enterprise data platform end to end on Databricks and AWS, setting the lakehouse architecture, medallion model, orchestration, and deployment patterns
• Set the standard for data modeling and structure, ensuring data is consistent, accurate, and reliable for the teams that depend on it
• Establish and enforce data governance: entitlements and access control, lineage, cataloging, and data quality SLAs
• Build the serving layer that delivers data to PMs and Analysts
• Partner with the investment teams as the platform’s primary users, along with Risk and Quant, keeping stakeholders aligned through clear communication, status, and planning
• Onboard and integrate new data sources across vendor, market, and alternative data
• Stay hands‑on as a player‑coach, working in the code and the data alongside the team
• Build and mentor a small, high‑performing data team, promoting engineering excellence and strong standards
• Make the firm’s data AI‑ready; structured and retrieval‑ready for applied AI across the firm
• Partner with the Head of Development to define clean interfaces between application services and the data platform

Key Objectives for the First 12–18 Months

• Take full ownership of the existing platform, ensuring stability, consistent delivery, and confidence from the investment, risk, and operations teams
• Publish documented data modeling standards and enforce automated data‑quality tests across all core datasets
• Enforce entitlements across all production datasets and deliver data lineage and a data‑quality SLA view the business can monitor
• Onboard the priority datasets from the roadmap, each delivered against a documented data contract and SLA
• Define a clear data roadmap that sequences new sources, masters, and products by business value and technical priority

Qualifications

• 10+ years of hands‑on data engineering experience, with recent ownership of production data platforms
• Proven success building and operating Databricks and AWS data platforms (Unity Catalog, Delta Lake, S3, serverless compute), with expert Python, SQL, and dbt in a medallion or equivalent tiered modeling approach
• Deep expertise in data modeling: dimensional and canonical modeling, security and reference masters, point‑in‑time data, and identifier resolution (FIGI, RIC, ISIN, CUSIP)
• Strong data governance background: entitlements, access control, lineage, cataloging, and data quality frameworks
• Strong architectural thinking, able to define clean data contracts and modernization paths
• High engineering standards and fluency in CI/CD, testing, and observability for data
• Experience leading small, high‑impact teams – player‑coach mentality preferred
• Strong communication skills and the ability to engage directly with business and investment stakeholders
• Prior hedge fund, asset management, or trading systems experience across market, reference, and alternative data highly desirable

Tech Stack

• Languages & Frameworks: Python, SQL, dbt
• Platform & Lakehouse: Databricks (Unity Catalog, Delta Lake, serverless compute), AWS (S3, Athena)
• Ingestion & Orchestration: Databricks Workflows, Airflow, managed connectors, web scraping
• Data Quality & Observability: dbt tests, Elementary, Datadog
• CI/CD & Version Control: Bitbucket Pipelines, contract and data testing frameworks
• Serving & Consumption: SQL endpoints/MCPs, Databricks dashboards and apps, Excel integration, email‑ready reporting