DataBricks Data Engineer

Globalchannelmanagement; New York (NY); On-site; USD 150,000 - 210,000; Full time

Job summary

Globalchannelmanagement is seeking a DataBricks Data Engineer in New York to lead end-to-end data solutions for Sales Incentive Compensation (SIC). The role requires 7+ years in data engineering, with 3+ years hands-on Databricks experience, and strong Python/SQL skills.

You will design, build, and optimize data pipelines that ingest, transform, and model incentive data, while integrating Databricks with cloud services and SIC platforms.

Qualifications

• Must have functional and technical experience with Sales Incentive Compensation (SIC) including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.
• At least 7 years of experience in data engineering, with a minimum of 3 years hands-on experience building and managing Databricks solutions.
• Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows.
• Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.
• Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.
• Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.
• Familiarity with SIC platforms (e.g., Anaplan, Oracle ICM, Varicent) and experience integrating them with data engineering platforms is a strong plus.
• Ability to understand complex incentive compensation rules and model them accurately in data pipelines and transformation logic.

Skills

Incentive compensation design

Python

SQL

Scala

ETL/ELT

Data modeling

Data ingestion

Pipeline orchestration

Databricks platform

Tools

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Azure Data Factory

Event Hubs

Kafka

Anaplan

Oracle ICM

Varicent

Job description

DataBricks Data Engineer requires:

• Must have functional and technical experience with Sales Incentive Compensation (SIC) including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.
• At least 7 years of experience in data engineering, with a minimum of 3 years hands‑on experience building and managing Databricks solutions.
• Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows.
• Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.
• Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.
• Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.
• Passion and ability to create holistic, end-to-end, integrated data solutions.
• Familiarity with SIC platforms (e.g., Anaplan, Oracle ICM, Varicent) and experience integrating them with data engineering platforms is a strong plus.
• Ability to understand complex incentive compensation rules and model them accurately in data pipelines and transformation logic.