Lead Data Engineer

Selby Jennings; New York (NY); On-site; USD 175,000 - 250,000; Full time

Job summary

Selby Jennings seeks a data platform engineer to drive cloud-based data platforms and pipelines for enterprise datasets. You will design scalable data models, leverage Python, SQL, Snowflake, and AWS, and mentor engineers while delivering production-grade data solutions.

Responsibilities include leading architecture decisions, improving data reliability, and collaborating with investment, research, and technology teams to push the platform forward.

Qualifications

• Experience building and supporting enterprise-scale data platforms and pipelines.
• Strong programming skills in Python and SQL.
• Expertise with Snowflake or a similar modern cloud data warehouse.
• Hands-on experience designing and deploying solutions within AWS.
• Experience with workflow orchestration tools such as Airflow.
• Knowledge of relational and NoSQL database technologies.
• Strong understanding of data modeling, data architecture, and large-scale data processing.
• Proven ability to lead technical initiatives and deliver complex projects.

Responsibilities

• Lead the development of cloud-based data platforms, pipelines, and data products leveraging Python, SQL, Snowflake, and AWS.
• Build scalable solutions for a variety of data sources, including market, alternative, and enterprise datasets.
• Enhance the firm's data platform through APIs, automation, metadata management, and emerging AI-driven capabilities.
• Improve data reliability through quality controls, monitoring, testing, and operational best practices.
• Troubleshoot complex data issues and implement long-term solutions to improve platform stability.
• Design efficient data models and datasets that support analytics, reporting, and quantitative research.
• Collaborate with investment, research, and technology teams to deliver scalable data solutions.
• Provide technical leadership through architecture guidance, code reviews, mentorship, and engineering best practices.
• Drive projects from design through production while influencing the future direction of the data platform.
• Opportunity to mentor and potentially lead a growing team of engineers.

Skills

Python

SQL

Snowflake

AWS

Data Modeling

Data Architecture

Leadership

NoSQL

Tools

Airflow

Job description

Salary: USD175000 - USD250000 per year

Responsibilities

• Lead the development of cloud-based data platforms, pipelines, and data products leveraging Python, SQL, Snowflake, and AWS.
• Build scalable solutions for a variety of data sources, including market, alternative, and enterprise datasets.
• Enhance the firm's data platform through APIs, automation, metadata management, and emerging AI-driven capabilities.
• Improve data reliability through quality controls, monitoring, testing, and operational best practices.
• Troubleshoot complex data issues and implement long-term solutions to improve platform stability.
• Design efficient data models and datasets that support analytics, reporting, and quantitative research.
• Collaborate with investment, research, and technology teams to deliver scalable data solutions.
• Provide technical leadership through architecture guidance, code reviews, mentorship, and engineering best practices.
• Drive projects from design through production while influencing the future direction of the data platform.
• Opportunity to mentor and potentially lead a growing team of engineers.

Qualifications

• Experience building and supporting enterprise-scale data platforms and pipelines.
• Strong programming skills in Python and SQL.
• Expertise with Snowflake or a similar modern cloud data warehouse.
• Hands-on experience designing and deploying solutions within AWS.
• Experience with workflow orchestration tools such as Airflow.
• Knowledge of relational and NoSQL database technologies.
• Strong understanding of data modeling, data architecture, and large-scale data processing.
• Proven ability to lead technical initiatives and deliver complex projects.

Preferred Qualifications

• Experience with Go or other backend development languages.
• Background in platform engineering or service-oriented architectures.
• Exposure to AI/LLM-powered data solutions and automation.
• Experience within financial services, investment management, trading, or market data environments.
• Track record of building highly reliable and scalable data systems.