Data Engineering Lead

Elevanta Partners; New York (NY); On-site; USD 150,000 - 200,000; Full time

Job summary

A Private Equity firm in New York is seeking an experienced Data Engineering Lead to architect and lead large-scale data initiatives. This hands-on role requires management of an offshore team and the implementation of robust data pipelines using Azure Databricks and PySpark. The ideal candidate will have over 15 years of experience in enterprise data programs and strong communication skills for stakeholder engagement. Preferred backgrounds include experience in Private Equity or Capital Markets data environments.

Qualifications

• 15+ years of experience delivering large-scale data programs in enterprise environments.
• Databricks Certified Data Engineer with 5+ years of experience leading Azure Databricks programs.
• Strong hands‑on experience with PySpark for data pipelines.

Responsibilities

• Lead and manage an offshore team of data engineers and testers.
• Architect and implement scalable data pipelines.
• Collaborate with stakeholders to align data solutions.

Skills

Data engineering leadership

PySpark

SQL

Data governance

Communication skills

Tools

Azure Databricks

Unity Catalog

Job description

Industry: Private Equity / Capital Markets

About the Company and the role

We are seeking an experienced Data Engineering Lead to drive large-scale data initiatives for a Private Equity firm based in New York. This is a hands‑on leadership role where you will architect, build, and lead modern data platforms on Azure Databricks while managing an offshore delivery team. You will work closely with business stakeholders, technology partners, and leadership to deliver high‑quality, scalable data solutions that support investment, reporting, and analytics use cases.

Responsibilities

• Act as the onsite lead and primary point of contact for data engineering initiatives
• Manage and mentor an offshore team of 5–7 data engineers and data testers, ensuring delivery quality and timelines
• Architect and implement robust, scalable data pipelines using PySpark and SQL
• Define and enforce best practices for data engineering, performance optimization, and data quality
• Work closely with data architects and stakeholders to align data solutions with enterprise data architecture
• Leverage Unity Catalog for data governance, security, and access control
• Collaborate with business users, analysts, and leadership to translate business requirements into technical solutions

Qualifications

• 15+ years of experience delivering large-scale data programs in enterprise environments
• Databricks Certified Data Engineer with 5+ years of experience leading Azure Databricks programs
• Strong hands‑on experience with PySpark, including 5+ years delivering production‑grade data pipelines
• Solid working knowledge of Unity Catalog and modern data governance practices
• Strong understanding of data architecture principles, including data lakes, lakehouse patterns, and ETL/ELT design
• Advanced SQL skills for data transformation, analysis, and optimization
• Proven experience managing offshore teams of data engineers and data testers
• Excellent communication skills with the ability to engage both technical and non‑technical stakeholders

Good to Have (Preferred Skills)

• Experience working on Private Equity or Capital Markets data programs
• Prior experience in regulated or high‑performance data environments