Data Engineering Lead - Azure Databricks & Global Delivery

Elevanta Partners; New York (NY); On-site; USD 150,000 - 200,000; Full time

Job summary

A Private Equity firm in New York is seeking an experienced Data Engineering Lead to architect and lead large-scale data initiatives. This hands-on role requires management of an offshore team and the implementation of robust data pipelines using Azure Databricks and PySpark. The ideal candidate will have over 15 years of experience in enterprise data programs and strong communication skills for stakeholder engagement. Preferred backgrounds include experience in Private Equity or Capital Markets data environments.

Qualifications

• 15+ years of experience delivering large-scale data programs in enterprise environments.
• Databricks Certified Data Engineer with 5+ years of experience leading Azure Databricks programs.
• Strong hands‑on experience with PySpark for data pipelines.

Responsibilities

• Lead and manage an offshore team of data engineers and testers.
• Architect and implement scalable data pipelines.
• Collaborate with stakeholders to align data solutions.

Skills

Data engineering leadership

PySpark

SQL

Data governance

Communication skills

Tools

Azure Databricks

Unity Catalog

Job description

A Private Equity firm in New York is seeking an experienced Data Engineering Lead to architect and lead large-scale data initiatives. This hands-on role requires management of an offshore team and the implementation of robust data pipelines using Azure Databricks and PySpark. The ideal candidate will have over 15 years of experience in enterprise data programs and strong communication skills for stakeholder engagement. Preferred backgrounds include experience in Private Equity or Capital Markets data environments.