Senior Databricks Data Engineer: Streaming & Delta Lake

HR POD - Hiring Talent Globally; Lahore; On-site; PKR 300,000 - 600,000; Full time

Job summary

HR POD - Hiring Talent Globally seeks a data engineer to own end-to-end ingestion pipelines using Databricks, Delta Lake and Medallion Architecture. You will work with Spark SQL, PySpark, and production-grade streaming data from Azure Event Hubs, Kafka, or Kinesis.

Responsibilities include handling schema drift, data quality, and coordinating with a US-based team to resolve incidents and ensure data correctness across Bronze-Silver-Gold layers.

Qualifications

• 4+ years of experience in data engineering with production Databricks experience.
• Strong expertise in Spark SQL, PySpark, Delta Lake, Medallion Architecture and Delta Live Tables (DLT).
• Hands-on with streaming ingestion using Azure Event Hubs, Kafka or Kinesis.
• Experience debugging data discrepancies across source-to-warehouse pipelines and handling schema drift.

Responsibilities

• Build and maintain resilient ingestion pipelines for third-party vendor REST APIs.
• Work with voice-AI observability and telephony data platforms.
• Handle multiple pagination schemes and manage rate limits with time-windowing.
• Implement robust schema-drift handling and alert when fields are renamed or removed.
• Own streaming ingestion from Azure Event Hubs into Databricks with Structured Streaming/Auto Loader.
• Manage checkpoints, offsets, watermarking and at-least-once deduplication.
• Develop Delta Lake pipelines using Medallion Architecture (Bronze-Silver-Gold).
• Build entity-resolution pipelines for dirty, free-text data.

Skills

SQL

Python

Data pipelines

Data quality

Tools

Databricks

Spark SQL

PySpark

Delta Lake

Delta Live Tables

Azure Event Hubs

Kafka

Kinesis

Holistics AML/AQL

dbt Metrics

Job description

HR POD - Hiring Talent Globally seeks a data engineer to own end-to-end ingestion pipelines using Databricks, Delta Lake and Medallion Architecture. You will work with Spark SQL, PySpark, and production-grade streaming data from Azure Event Hubs, Kafka, or Kinesis.

Responsibilities include handling schema drift, data quality, and coordinating with a US-based team to resolve incidents and ensure data correctness across Bronze-Silver-Gold layers.