Senior Data Engineer - Real-Time Spark/PySpark Pipelines

Recruitzz; Lahore; On-site; PKR 3,200,000 - 5,600,000; Full time

Job summary

Recruitzz in Lahore, Pakistan, is seeking a Senior Data Engineer to own end-to-end data pipelines using Databricks and Python, on-site near FC College Lahore.

The role requires 5+ years in data engineering with production experience in Spark SQL, PySpark, Delta Lake, and medallion architecture; you will work on streaming data ingestion, API integration, and data matching, collaborating with a US team.

Experience with a semantic layer and strong English communication is essential.

Qualifications

• 5+ years in data engineering with substantial production Databricks experience: Spark SQL, PySpark, Delta Lake, medallion architecture, and DLT.
• Structured Streaming or equivalent streaming ingestion in production (Event Hubs, Kafka, or Kinesis sources), including checkpoint recovery, watermarking, and dedup strategies.
• Demonstrated experience debugging source-vs-warehouse discrepancies — you can walk through a real incident where counts didn't match and explain how you found the cause.
• Production integration with third-party REST APIs, including pagination edge cases, rate/row limits, retries, and schema-drift handling.
• Entity resolution or data-matching work on messy real-world text data.
• Experience with a metrics/semantic layer (Holistics AML/AQL, dbt metrics, or LookML) and a working understanding of why non-additive measures can't be computed from pre-aggregated rollups.
• Strong SQL and Python; comfortable owning pipelines end-to-end without heavy oversight.
• Written and spoken English strong enough for async collaboration with a US team.

Responsibilities

Skills

Databricks

Spark SQL

PySpark

Delta Lake

Medallion architecture

DLT

Structured Streaming

Streaming ingestion

Event Hubs

Kafka

Kinesis

Checkpointing

Watermarking

Deduplication

Source-vs-Warehouse

REST APIs

Pagination

Rate limits

Retries

Schema drift

Entity resolution

Data matching

Holistics AML/AQL

dbt metrics

LookML

SQL

Python

End-to-end pipelines

English

Tools

dbt

LookML

Job description

Recruitzz in Lahore, Pakistan, is seeking a Senior Data Engineer to own end-to-end data pipelines using Databricks and Python, on-site near FC College Lahore.

The role requires 5+ years in data engineering with production experience in Spark SQL, PySpark, Delta Lake, and medallion architecture; you will work on streaming data ingestion, API integration, and data matching, collaborating with a US team.

Experience with a semantic layer and strong English communication is essential.