Big Data Developer

Russell Tobin; New York (NY); Hybrid; USD 103,320 - 107,452; Full time

Job summary

Russell Tobin is seeking a seasoned Databricks Data Engineer in New York, NY. The role focuses on designing and implementing data pipelines on Databricks to support ingestion, transformation, and analytics for sales incentive processing and enterprise data workflows.

You will work hybrid in NYC, with a six-month contract and potential extension, earning $75–78 per hour on W-2, DOE. The position requires 7+ years in data engineering, strong SQL, Python/Scala, and experience with SIC systems and

Qualifications

• At least 7 years in data engineering with 3+ years on Databricks.
• Deep knowledge of Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows.
• Strong Python/Scala and SQL skills.
• Experience in ingestion, transformation, modeling and pipelines.
• Experience with SIC systems and design.
• Familiarity with SIC platforms and integration with data platforms is a plus.
• Ability to translate complex business rules into scalable pipelines.
• Experience with cloud data sources and platforms (Azure/AWS/GCP).

Responsibilities

• Design, develop and implement Databricks data pipelines.
• Support data ingestion, transformation and analytics.
• Collaborate with stakeholders to translate requirements into architectures.
• Ensure data quality, performance and end-user data experiences.

Skills

Databricks

Python

SQL

Scala

ETL/ELT

Data Modeling

Data Ingestion

Workflow Orchestration

SIC Knowledge

Stakeholder Mgmt

Cloud Integration

Tools

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Azure Data Factory

Event Hubs

Kafka

Anaplan (SIC)

Oracle ICM

Varicent

Job description

Contract (6 Months + Possible Extension)

New York, NY (Hybrid)

Pay Rate: $75-78 an hour on W2 (DOE)

Job Description:

We are seeking a skilled and experienced Databricks Data Engineer to join our team. As a Databricks Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions using the Databricks platform to support data ingestion, transformation, performance reporting, analytics, sales incentive compensation processing, and enterprise data pipeline processes.

Requirements:

• At least 7 years of experience in data engineering, with a minimum of 3 years hands-on experience building and managing Databricks solutions.
• Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows.
• Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.
• Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.
• Functional and technical experience with Sales Incentive Compensation (SIC) — including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.
• Familiarity with SIC platforms (e.g., Anaplan, Oracle ICM, Varicent) and experience integrating them with data engineering platforms is a strong plus.
• Ability to understand complex incentive compensation rules and model them accurately in data pipelines and transformation logic.
• Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.
• Passion and ability to create holistic, end-to-end, integrated data solutions.
• Stakeholder management skills; ability to challenge constructively and translate complex business requirements into scalable data architectures — not just gather requirements.
• Strong analytical and problem-solving skills, with the ability to understand complex data requirements and implement reliable, efficient solutions.
• Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders.
• Must work well in a team environment while being a resourceful, independent self-starter able to work effectively with minimal direction.
• Skilled at documenting complex data architectures and pipelines in a professional manner.
• Ability to work independently and manage multiple tasks and priorities in a fast-paced environment.
• Passion for data quality, performance optimization, and delivering excellent end-user data experiences.
• Databricks Certified Data Engineer Associate or Professional certification is a plus.