Senior Data Engineer

TalentHue- Careers; Lahore; On-site; PKR 2,400,000 - 4,200,000; Full time

Job summary

Senior Data Engineer at TalentHue- Careers in Lahore, Pakistan, responsible for building end-to-end data pipelines, ingestion, transformation, and orchestration across the lakehouse stack. You will deploy through CI/CD onto containerized infrastructure.

Role requires deep experience with Airflow, Kafka, Spark, and Informatica IDMC, plus Docker/Kubernetes-based deployments, data quality checks, and SLA monitoring. On-site, full-time position with strong collaboration with data teams.

Qualifications

• 6+ years of hands-on data engineering experience building production pipelines.
• Strong working knowledge of Apache Airflow for orchestration, DAG design, sensors, and operational troubleshooting.
• Production experience with Kafka, producers/consumers, schema registry, partitioning strategy.
• Direct experience with Informatica IDMC (or PowerCenter/IICS background actively transitioning to IDMC) for managed ETL/ELT.
• Comfort with GitLab CI/CD pipelines and Docker/Kubernetes-based deployment of data workloads.
• Strong Python and SQL skills; ability to read/write Spark (PySpark) jobs independently.

Responsibilities

• Builds and operates end-to-end data pipelines, ingestion, transformation, and orchestration across lakehouse stack.
• Collaborate with Data Modeler and Data Architect to ensure pipeline output matches target schema and table-format requirements.
• Troubleshoot production pipeline issues and participate in on-call/SLA support rotations as needed.

Skills

Airflow

Kafka

Spark

Python

SQL

DAG design

Data quality

SLA monitoring

Iceberg/Delta/Hudi

Informatica IDMC

GitLab CI/CD

Docker/Kubernetes

PySpark

Open formats

Tools

Informatica IDMC

Docker

Kubernetes

GitLab CI/CD

Job description

Senior Data Engineer

A technology-driven software company delivering tailored digital solutions to startups, SMEs, and large enterprises. Their expertise spans product development, system integration, and scalable software engineering.

Role Introduction

Builds and operates end-to-end data pipelines, ingestion, transformation, and orchestration, across the lakehouse stack, deploying through CI/CD onto containerized infrastructure.

Features

• Onsite
• Fulltime

Requirements

• Design and build batch and streaming ingestion pipelines using Airflow, Kafka, and Spark.
• Develop transformation logic and ETL/ELT workflows using Informatica IDMC alongside custom Spark jobs where needed.
• Containerize pipeline code and deploy via GitLab CI/CD onto Dockerized/Kubernetes infrastructure.
• Write and maintain Airflow DAGs with proper dependency management, retries, and SLA monitoring.
• Implement data quality checks and validation logic at each stage of the pipeline.
• Optimize Spark jobs for performance and cost (partitioning, caching, shuffle management) writing into Iceberg tables.
• Collaborate with the Data Modeler and Data Architect to ensure pipeline output matches target schema and table-format requirements.
• Troubleshoot production pipeline issues and participate in on-call/SLA support rotations as needed.

Specifications

• 6+ years of hands-on data engineering experience building production pipelines, ideally with 3+ years specifically on Spark.
• Strong working knowledge of Apache Airflow for orchestration, DAG design, sensors, and operational troubleshooting.
• Production experience with Kafka, producers/consumers, schema registry, partitioning strategy.
• Direct experience with Informatica IDMC (or PowerCenter/IICS background actively transitioning to IDMC) for managed ETL/ELT.
• Comfort with GitLab CI/CD pipelines and Docker/Kubernetes-based deployment of data workloads.
• Strong Python and SQL skills; ability to read/write Spark (PySpark) jobs independently.
• Experience writing to and reading from open table formats (Iceberg, Delta Lake, or Hudi) in a lakehouse setup.

Expertise

Skills: Airflow, Docker/Kubernetes, GitLab CI/CD, Informatica IDMC, Kafka, Spark