DataBricks Data Engineer - 14993-1

Cephas Consultancy Services Private Limited; New York (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Cephas Consultancy Services Private Limited in New York, NY is seeking a Databricks Data Engineer to design, develop, and implement data pipelines and dashboards on the Databricks platform. The role focuses on ingestion, transformation, performance reporting, and enterprise data workflows.

You will own the Databricks architecture and collaborate with Finance, IT, and BI teams to ensure integrated, auditable data solutions including SIC processing and payout reporting.

Qualifications

• 7+ years in data engineering with 3+ years in Databricks solutions.
• Strong Python/Scala and SQL proficiency; ETL/ELT skills.
• Experience with SIC data, incentive plans, and related rules.
• Ability to design end-to-end data pipelines and dashboards.

Responsibilities

• Design and build Databricks data pipelines, Delta Lake models, and notebooks.
• Own Databricks architecture for scalable, integrated workflows.
• Develop SIC data processing: quotas, attainment, commissions, payouts.
• Collaborate with Finance/Sales Ops to model incentive rules.
• Test, debug, optimize pipelines; provide user support and documentation.

Skills

Python

Scala

SQL

ETL/ELT

Data modeling

Data pipelines

SIC knowledge

Cloud integration

Cross-functional collaboration

Problem solving

Tools

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Azure Data Factory

Event Hubs

Kafka

Job description

DataBricks Data Engineer - 14993-1

Cephas Consultancy Services Private Limited | New York, New York, United States

About this position

DataBricks Data Engineer - Contract (CLI-45623-7)

We are seeking a skilled and experienced Databricks Data Engineer to join our team. As a Databricks Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions using the Databricks platform to support data ingestion, transformation, performance reporting, analytics, sales incentive compensation processing, and enterprise data pipeline processes.

Responsibilities

• Collaborate with business stakeholders to understand their data requirements and translate them into scalable technical solutions.
• Design and develop Databricks-based data pipelines, including data models, Delta Lake tables, and dashboards/notebooks.
• Responsible for the overall Databricks architecture to ensure new pipelines and enhancements to existing workflows are designed in an integrated and optimal manner.
• Maintain and optimize all current Databricks workspaces and pipelines; capture and deliver requirements for new or improved functionality to achieve further automation and integration of data and processes.
• Design and implement data pipelines to support Sales Incentive Compensation (SIC) processes, including quota management, attainment calculations, commission processing, and payout reporting.
• Partner with Sales Operations and Finance teams to understand incentive plan rules and translate them into accurate, auditable data transformations and models.
• Perform testing and debugging of data pipelines and notebooks to identify and resolve issues and ensure optimal performance.
• Provide technical support and troubleshooting for Databricks-based solutions, working closely with end‑users and analysts to address their needs and resolve any data‑related problems.
• Stay up to date with Databricks features and enhancements (Unity Catalog, Delta Live Tables, Photon, etc.), and proactively identify opportunities to leverage them to improve data quality, performance, and analytics capabilities.
• Collaborate with cross‑functional teams, including finance, IT, and business intelligence teams, to ensure seamless integration and alignment of Databricks with other systems and processes.
• Document technical specifications, data models, pipeline configurations, and development processes to ensure knowledge transfer and maintain system documentation.

Requirements

• At least 7 years of experience in data engineering, with a minimum of 3 years hands‑on experience building and managing Databricks solutions.
• Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows.
• Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.
• Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.
• Functional and technical experience with Sales Incentive Compensation (SIC) — including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.
• Familiarity with SIC platforms (e.g., Anaplan, Oracle ICM, Varicent) and experience integrating them with data engineering platforms is a strong plus.
• Ability to understand complex incentive compensation rules and model them accurately in data pipelines and transformation logic.
• Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.
• Passion and ability to create holistic, end‑to‑end, integrated data solutions.
• Stakeholder management skills; ability to challenge constructively and translate complex business requirements into scalable data architectures — not just gather requirements.
• Strong analytical and problem‑solving skills, with the ability to understand complex data requirements and implement reliable, efficient solutions.
• Excellent communication and collaboration skills, with the ability to work effectively with cross‑functional teams and stakeholders.
• Must work well in a team environment while being a resourceful, independent self‑starter able to work effectively with minimal direction.
• Skilled at documenting complex data architectures and pipelines in a professional manner.
• Ability to work independently and manage multiple tasks and priorities in a fast‑paced environment.
• Passion for data quality, performance optimization, and delivering excellent end‑user data experiences.
• Databricks Certified Data Engineer Associate or Professional certification is a plus.

Equity, diversity, and inclusion. Equal opportunity employer. No discrimination on the basis of race, color, religion, sex, age, national origin, disability, or other protected characteristic.