Senior Databricks Data Engineer: Delta Lake & SIC Pipelines

Talentify; New York (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

Talentify is seeking a Databricks Data Engineer to design, develop, and implement data engineering solutions on the Databricks platform to support ingestion, transformation, reporting, analytics, and SIC processing.

You will collaborate with finance, IT, and BI teams to ensure seamless integration and high-quality data, while staying up-to-date with Unity Catalog and Delta Live Tables to optimize performance.

Qualifications

• 7+ years in data engineering with 3+ years building Databricks solutions.
• Experience with Delta Lake, Unity Catalog, Delta Live Tables, and Databricks Workflows.
• Proficient in Python and/or Scala; SQL expertise required.
• Experience designing data ingestion, transformation, modeling, and orchestration pipelines.
• Familiarity with Sales Incentive Compensation (SIC) platforms and integration.
• Experience integrating Databricks with cloud platforms and data sources.

Responsibilities

• Design and develop Databricks-based data pipelines and data models.
• Maintain and optimize Databricks workspaces and pipelines; implement enhancements.
• Collaborate with stakeholders to translate data requirements into scalable solutions.
• Implement SIC data pipelines including quota, attainment, and payout reporting.
• Provide debugging and support for Databricks-based solutions.
• Stay current with Databricks features to improve data quality and performance.

Skills

Databricks

Python

SQL

Scala

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Cloud platforms (Azure/AWS/GCP)

SIC data pipelines

Stakeholder management

Communication

Tools

Azure Data Factory

Kafka

Job description

Talentify is seeking a Databricks Data Engineer to design, develop, and implement data engineering solutions on the Databricks platform to support ingestion, transformation, reporting, analytics, and SIC processing.

You will collaborate with finance, IT, and BI teams to ensure seamless integration and high-quality data, while staying up-to-date with Unity Catalog and Delta Live Tables to optimize performance.