Databricks Data Engineer (Hybrid - NYC)

The Mom Project; New York (NY); Hybrid; USD 123,984 - 206,640; Full time

Benefits offered by this job

Medical

Dental

Vision

401k

Job summary

The Mom Project is seeking a Databricks Data Engineer on a contract basis to design, develop, and implement data pipelines and analytics using Databricks. This role is hybrid (3 days on-site, 2 days remote per week) in New York City and involves collaboration with Finance, IT, BI, and Sales Ops teams to support incentive compensation processing and enterprise data pipelines.

Ideal candidates will have 7+ years in data engineering, 3+ years directly building Databricks solutions, and strong

Qualifications

• 7+ years of data engineering experience.
• Minimum 3 years building and supporting Databricks solutions.
• Experience with Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.

Responsibilities

• Collaborate with stakeholders to capture requirements and design scalable data engineering solutions.
• Build and maintain Databricks-based data pipelines, data models, Delta Lake tables, notebooks, and dashboards.
• Define and maintain Databricks architecture to support integrated, scalable data solutions.

Skills

Databricks data engineering

Python

SQL

Stakeholder management

Problem solving

Tools

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Azure Data Factory

Event Hubs

Kafka

Job description

With roots dating back to the 1800s, our Customer is a global professional services firm HQ’d in New York City with businesses in insurance brokerage, risk management, reinsurance services, talent management, investment advisory, and management consulting. Today, they are a unique group of businesses with many enduring competitive advantages. Among them are the quality of their colleagues, deep client relationships, vast global footprint, depth of intellectual capital, and cohesive and collaborative culture.

We are seeking a Databricks Data Engineer on a contract basis to support our Customer’s business needs. This role is hybrid (3 days on-site, 2 days remote/week) in NYC. As a Databricks Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions using the Databricks platform to support data ingestion, transformation, performance reporting, analytics, sales incentive compensation processing, and enterprise data pipeline processes.

Responsibilities

• Collaborate with business stakeholders to gather requirements and translate them into scalable data engineering solutions
• Design, develop, and maintain Databricks-based data pipelines, data models, Delta Lake tables, notebooks, and dashboards
• Define and maintain the overall Databricks architecture to support integrated, scalable, and optimized data solutions
• Enhance and optimize existing Databricks workspaces, pipelines, and workflows
• Design and implement data pipelines supporting Sales Incentive Compensation (SIC) processes, including quota management, attainment tracking, commission calculations, and payout reporting
• Partner with Sales Operations and Finance teams to translate incentive compensation rules into accurate data models and transformations
• Perform testing, debugging, and performance tuning of data pipelines and notebooks
• Provide technical support and troubleshoot Databricks-based solutions
• Evaluate and implement new Databricks features such as Unity Catalog, Delta Live Tables, Photon, and other platform enhancements
• Integrate Databricks with cloud platforms and enterprise data sources
• Collaborate with Finance, IT, Business Intelligence, and other cross-functional teams to deliver end-to-end data solutions
• Document data architectures, pipeline configurations, technical specifications, and development processes

Skills And Qualifications

• 7+ years of experience in data engineering
• Minimum of 3 years of hands‑on experience building and supporting Databricks solutions
• Extensive experience with Databricks components, including Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows
• Strong proficiency in Python and/or Scala
• Advanced SQL skills
• Experience with data ingestion, ETL/ELT development, data modeling, and pipeline orchestration
• Experience designing and supporting Sales Incentive Compensation (SIC) data solutions
• Knowledge of incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing
• Experience integrating Databricks with Azure, AWS, or Google Cloud Platform
• Experience with Azure Data Factory, Event Hubs, Kafka, or similar data integration technologies
• Strong stakeholder management skills with the ability to translate complex business requirements into scalable technical solutions
• Strong analytical, troubleshooting, and problem‑solving skills
• Excellent written and verbal communication skills
• Experience documenting technical architectures, pipelines, and data models
• Ability to manage multiple priorities and work independently in a fast‑paced environment
• Passion for data quality, performance optimization, and scalable data architecture

Preferred Qualifications

• Experience with Sales Incentive Compensation platforms such as Anaplan, Oracle ICM, or Varicent
• Experience integrating incentive compensation platforms with enterprise data engineering solutions
• Databricks Certified Data Engineer Associate or Professional certification

We offer a competitive salary range for this position. Most candidates who join our team are hired at the median of this range, ensuring fair and equitable compensation based on experience and qualifications.

Contractor benefits are available through our 3rd Party Employer of Record (Available upon completion of waiting period for eligible engagements). Benefits include: Medical, Dental, Vision, 401k.

An Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.