DataBricks Data Engineer

Overture Partners; New York (NY); On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

Medical plans

401(k) starting day one

Voluntary benefits (life, disability)

Sick time

Job summary

Overture Partners is seeking an experienced Databricks Data Engineer to join a growing data and analytics team. You will design scalable data solutions powering reporting, analytics, and sales compensation with deep Databricks expertise.

You will collaborate across Finance, Sales Operations, BI, and IT to translate complex business rules into robust data architectures, automate workflows, and optimize performance.

Qualifications

• 7+ years of professional experience in data engineering, including at least 3 years of hands-on Databricks development.
• Strong expertise with Databricks components (Delta Lake, Delta Live Tables, Unity Catalog, Databricks Workflows).
• Advanced proficiency in SQL and Python or Scala for data engineering and pipeline development.
• Extensive experience building scalable ETL/ELT pipelines, data models, and enterprise data architectures.
• Functional and technical knowledge of Sales Incentive Compensation (SIC) processes.
• Experience integrating cloud-based data platforms in Azure, AWS, or Google Cloud.
• Strong analytical, troubleshooting, and performance optimization skills.
• Excellent communication skills for collaboration across teams.

Responsibilities

• Design, develop, and maintain scalable data pipelines using the Databricks platform.
• Build and optimize Delta Lake tables, notebooks, workflows, and data models for enterprise analytics.
• Architect integrated data solutions for long-term scalability and performance.
• Enhance Databricks workspaces by improving automation and reliability.
• Develop data pipelines supporting Sales Incentive Compensation processes, including quota and payout reporting.
• Collaborate with Sales Operations, Finance, BI, and tech teams to translate business rules into data transformations.
• Test, validate, debug, and tune pipelines to ensure data accuracy.
• Integrate Databricks with cloud platforms and streaming/ingestion technologies.
• Evaluate Unity Catalog, Delta Live Tables, and Photon to improve governance and data quality.
• Produce technical documentation covering architecture, data models, and standards.
• Provide production support and promote data engineering best practices.

Tools

Databricks platform

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

SQL

Python/Scala

ETL/ELT pipelines

Sales Incentive Compensation (SIC)

Cloud platforms (Azure/AWS/GC)

Databricks Photon

Job description

Due to client requirements, applicants must be able to work on a w2 basis

Job Overview

We are seeking an experienced Databricks Data Engineer to join a growing data and analytics team responsible for building scalable, high-performance data solutions. This role is ideal for a hands‑on engineer with deep expertise in Databricks, modern data engineering practices, and Sales Incentive Compensation (SIC) processes. Candidates with a proven track record of designing enterprise data pipelines, collaborating with cross‑functional stakeholders, and delivering reliable, business‑critical data solutions will be prioritized for interviews.

Must Haves

• 7+ years of professional experience in data engineering, including at least 3 years of hands‑on Databricks development.
• Strong expertise with Databricks components, including Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
• Advanced proficiency in SQL and Python or Scala for data engineering and pipeline development.
• Extensive experience building scalable ETL/ELT pipelines, data models, and enterprise data architectures.
• Functional and technical knowledge of Sales Incentive Compensation (SIC), including quota management, attainment calculations, commission processing, and payout reporting.
• Experience integrating cloud‑based data platforms and services within Azure, AWS, or Google Cloud environments.
• Strong analytical, troubleshooting, and performance optimization skills.
• Excellent communication skills with the ability to collaborate effectively across technical and business teams.

What the Client Needs You to Do

In this role, you will design and optimize enterprise data solutions that power reporting, analytics, and sales compensation processes. You will work closely with business stakeholders, finance teams, sales operations, and technology partners to transform complex business requirements into scalable, secure, and maintainable data architectures. Success in this position requires a proactive mindset, strong ownership of data quality, and the ability to continuously improve platform performance through automation and best practices.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using the Databricks platform.
• Build and optimize Delta Lake tables, notebooks, workflows, and supporting data models for enterprise analytics.
• Architect integrated data solutions that support long‑term scalability, maintainability, and performance.
• Enhance existing Databricks workspaces by improving automation, reliability, and operational efficiency.
• Develop data pipelines supporting Sales Incentive Compensation processes, including quota allocation, commission calculations, attainment tracking, and payout reporting.
• Partner with Sales Operations, Finance, Business Intelligence, and technical teams to translate business rules into accurate and auditable data transformations.
• Perform testing, validation, debugging, and performance tuning to ensure data accuracy and pipeline reliability.
• Integrate Databricks with cloud platforms and enterprise data sources, including modern streaming and ingestion technologies.
• Evaluate and implement new Databricks capabilities such as Unity Catalog, Delta Live Tables, and Photon to improve governance, performance, and data quality.
• Produce clear technical documentation covering architecture, data models, pipeline configurations, and development standards.
• Provide technical guidance and production support while troubleshooting complex data issues across the platform.
• Promote data engineering best practices, governance standards, and continuous improvement across the data ecosystem.

Additional Information

• This position works closely with cross‑functional teams including Data Engineering, Finance, Sales Operations, Information Technology, and Business Intelligence.
• Experience with Sales Incentive Compensation platforms such as Anaplan, Oracle Incentive Compensation Management (ICM), Varicent, or comparable solutions is highly desirable.
• Candidates should demonstrate the ability to interpret complex business logic and translate it into scalable, maintainable data engineering solutions.
• A collaborative mindset combined with the ability to work independently and manage multiple priorities is essential.
• Experience with cloud‑native data services such as Azure Data Factory, Event Hubs, Kafka, or equivalent technologies is preferred.
• A passion for building reliable, high‑quality data platforms and delivering exceptional user experiences is highly valued.
• Databricks Certified Data Engineer Associate or Professional certification is considered an advantage but is not required.

W2 employees of Overture Partners who work 30 or more hours per week are eligible for the following benefits: medical (choice of 3 plans), 401(k) starting on day one, a variety of voluntary benefits including life and disability insurance, and sick time if required by law in the worked‑in state/locality.