Lead Data Engineer - Real-Time Pipelines & Lakehouse

Systems Limited; Punjab; On-site; PKR 3,200,000 - 5,200,000; Full time

Job summary

Systems Limited is seeking a Principal Data Engineer with 8+ years of hands-on experience in real-time and batch data processing. You will design and maintain scalable pipelines, containerize applications with Docker, and orchestrate workflows using Airflow or Azure Data Factory.

Experience with Spark (PySpark), Kafka, Flink, and lakehouse storage is essential. Leadership and agile collaboration are required to drive robust data solutions.

Qualifications

• 8+ years of hands-on data engineering experience with real-time and batch pipelines.
• Strong Python coding practices with modular, testable code.
• Experience with at least one cloud platform (Azure preferred).
• Real-time data processing using Kafka, Flink or Spark Streaming.
• Spark (PySpark) expertise for batch processing.
• Lakehouse architectures and columnar storage (Parquet/Delta/Iceberg).
• Orchestration using Airflow or Azure Data Factory.
• APIs to expose large datasets; data quality tooling (Great Expectations).
• Docker, Kubernetes and Helm for deployments; Terraform for infra.
• Agile mindset; mentoring data engineers; collaboration with DevOps.

Responsibilities

• Design, build, and maintain real-time data pipelines using Kafka, Flink, or Spark Streaming.
• Develop batch workflows with Apache Spark (PySpark).
• Orchestrate data pipelines with Airflow or Azure Data Factory.
• Containerize apps with Docker; deploy with Helm on Kubernetes.
• Use Lakehouse formats (Parquet/Delta/Iceberg) for storage.
• Build high-performance analytics engines with Trino or Presto.
• Collaborate with DevOps to manage infra with Terraform and CI/CD via Azure DevOps.
• Ensure data quality with Great Expectations; write modular Python/SQL.
• Develop observability to monitor pipeline performance; optimize throughput.
• Lead and mentor a team of data engineers; participate in agile ceremonies.

Education

Bachelor's degree in computer science or related discipline

Tools

Docker

Kubernetes

Helm

Apache Airflow

Azure Data Factory

Terraform

Azure DevOps

Trino/Presto

Spark (PySpark)

Kafka

Flink

Delta Lake

Parquet

Iceberg

ClickHouse

Job description

Systems Limited is seeking a Principal Data Engineer with 8+ years of hands-on experience in real-time and batch data processing. You will design and maintain scalable pipelines, containerize applications with Docker, and orchestrate workflows using Airflow or Azure Data Factory.

Experience with Spark (PySpark), Kafka, Flink, and lakehouse storage is essential. Leadership and agile collaboration are required to drive robust data solutions.