Lead Data Engineer (DataOps) | Azure, Databricks, CI/CD

Harvey Nash; New York (NY); Hybrid; USD 140,000 - 160,000; Full time

Job summary

Harvey Nash seeks a DataOps leader to build and scale our data operations from the ground up. You’ll drive CI/CD, standards, automation, and operational discipline across Data Engineering, DevOps, Enterprise Architecture, Security, and BI teams.

The role centers on Azure and Databricks, including SQL, Terraform, and on-prem/cloud data platforms. You’ll be a hands-on technical leader who shapes how data engineering is delivered and operated, while mentoring engineers and enabling teams to deliver

Qualifications

• 8+ years in data engineering, analytics engineering, or platform engineering.
• 3+ years in a lead/senior engineering or data platform role.
• Strong experience with Azure and Databricks/Lakehouse environments.
• Hands-on experience with SQL, Python/PySpark, data pipelines, and orchestration.
• Strong understanding of CI/CD and Azure DevOps.

Responsibilities

• Build and evolve the DataOps operating model, standards, and best practices.
• Lead CI/CD, automation, orchestration, monitoring, and data quality initiatives.
• Partner with Data Engineering, Architecture, Analytics, AI/ML, DevOps, Security teams.
• Improve reliability, scalability, performance, and operational consistency of data platforms.
• Drive cloud cost optimization and operational excellence; mentor engineers and shape the DataOps organization.

Skills

Azure

Databricks/Lakehouse

SQL

Python/PySpark

Data pipelines

Orchestration

CI/CD

Azure DevOps

Tools

ADF

Airflow

Dagster

dbt

Terraform

GitHub Actions

GitLab

Job description

Location: Short Hills, NJ / Hartford CT / Charlotte, NC (Hybrid working - 3 days a week)

Perm/FTE Role

US citizens and Green Card Holders and those authorized to work in the US are encouraged to apply. We are unable to sponsor H1b candidates at this time

Job summary

• This is a brand-new DataOps leadership role created to help build and formalize DataOps function from the ground up. With major investments in cloud data, analytics, and AI, the company needs someone who can bring greater structure, consistency, automation, and operational discipline to its growing data environment.
• You’ll be a hands-on technical leader working across Data Engineering, DevOps, Enterprise Architecture, Information Management, Security, Infrastructure, and BI teams. The focus is not simply managing people, but building the DataOps practice, establishing CI/CD and engineering standards, improving pipelines, and helping teams operationalize data solutions effectively.
• The environment is centered around Azure and Databricks, with technologies including SQL, Terraform, Azure DevOps, ETL tools, and on-prem/cloud data platforms.
• This role is ideal for a self-starter who can wear multiple hats: part DataOps leader, part architect, part hands-on practitioner, and part change agent. Strong communication and collaboration skills are equally important as technical depth.
• In short: you’ll have the opportunity to build the DataOps function from the ground up and shape how data engineering is delivered and operated across the organization.

What You’ll Do

• Build and evolve the DataOps operating model, standards, and best practices.
• Lead CI/CD, automation, orchestration, monitoring, and data quality initiatives.
• Partner with Data Engineering, Architecture, Analytics, AI/ML, DevOps, Security, and Infrastructure teams.
• Improve reliability, scalability, performance, and operational consistency of data platforms.
• Drive cloud cost optimization and operational excellence.Provide technical leadership, mentor engineers, and help shape the future DataOps organization.

What We’re Looking For

• 8+ years in data engineering, analytics engineering, or platform engineering.
• 3+ years in a lead/senior engineering or data platform role.
• Strong experience with Azure and Databricks/Lakehouse environments.
• Hands-on experience with SQL, Python/PySpark, data pipelines, and orchestration.
• Strong understanding of CI/CD and Azure DevOps.
• Experience with tools such as ADF, Airflow, Dagster, dbt, Terraform, GitHub Actions, or GitLab.
• Strong communication, collaboration, and technical leadership skills.
• Experience in insurance, financial services, healthcare, or other regulated environments is a plus.

If you enjoy building DataOps practices from the ground up and want to have a direct impact on how enterprise data is delivered and operated, we'd like to hear from you.

A reasonable, good faith estimate of the minimum and maximum base salary for this position is $140k to $160K . This position will also include a discretionary bonus or 10% that generally will be depending on a variety of factors. Employee benefits will also be available, and details are available like Pension/401K/ Paid Vacation/ Life, Medical & Dental insurance etc.

* The pay range listed above reflects the expected starting salary /Pay rate for this role. This range may be adjusted based on market conditions, location, and other relevant factors. The Company will determine the final starting salary/Pay rate in consultation with the selected candidate(s), in full compliance with applicable laws