Databricks Data Engineer: End-to-End Pipelines & SIC

Cephas Consultancy Services Private Limited; New York (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Cephas Consultancy Services Private Limited in New York, NY is seeking a Databricks Data Engineer to design, develop, and implement data pipelines and dashboards on the Databricks platform. The role focuses on ingestion, transformation, performance reporting, and enterprise data workflows.

You will own the Databricks architecture and collaborate with Finance, IT, and BI teams to ensure integrated, auditable data solutions including SIC processing and payout reporting.

Qualifications

• 7+ years in data engineering with 3+ years in Databricks solutions.
• Strong Python/Scala and SQL proficiency; ETL/ELT skills.
• Experience with SIC data, incentive plans, and related rules.
• Ability to design end-to-end data pipelines and dashboards.

Responsibilities

• Design and build Databricks data pipelines, Delta Lake models, and notebooks.
• Own Databricks architecture for scalable, integrated workflows.
• Develop SIC data processing: quotas, attainment, commissions, payouts.
• Collaborate with Finance/Sales Ops to model incentive rules.
• Test, debug, optimize pipelines; provide user support and documentation.

Skills

Python

Scala

SQL

ETL/ELT

Data modeling

Data pipelines

SIC knowledge

Cloud integration

Cross-functional collaboration

Problem solving

Tools

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Azure Data Factory

Event Hubs

Kafka

Job description

Cephas Consultancy Services Private Limited in New York, NY is seeking a Databricks Data Engineer to design, develop, and implement data pipelines and dashboards on the Databricks platform. The role focuses on ingestion, transformation, performance reporting, and enterprise data workflows.

You will own the Databricks architecture and collaborate with Finance, IT, and BI teams to ensure integrated, auditable data solutions including SIC processing and payout reporting.