Senior Data Engineer

Systems Limited; Punjab; On-site; PKR 2,400,000 - 3,600,000; Full time

Job summary

Systems Limited is seeking a Senior Data Engineer with 6+ years of hands-on experience to design and maintain scalable data pipelines. You will work with real-time streaming (Kafka, Flink, Spark Streaming) and batch processing (Spark PySpark), while containerizing applications and orchestrating with Airflow or Azure Data Factory.

You will mentor a team, implement lakehouse storage formats, and ensure data quality with Great Expectations in an agile setting.

Qualifications

• Proficiency in Python with modular, testable code.
• Strong SQL skills and large-scale data handling experience.
• Hands-on cloud experience, preferably Azure.
• Real-time data processing with Kafka, Flink, or Spark Streaming.
• Spark (PySpark) for batch processing.
• Experience with lakehouse architectures and columnar storage.
• Orchestration with Airflow or Azure Data Factory.
• APIs to expose large datasets.
• Docker, Kubernetes, Helm in production environments.
• Experience with data quality tooling (Great Expectations).
• Terraform for infra provisioning, basic usage.

Responsibilities

• Design and build real-time data pipelines using streaming frameworks.
• Develop batch workflows with Spark (PySpark).
• Orchestrate data jobs with Airflow or Azure Data Factory.
• Containerize apps with Docker; deploy via Helm on Kubernetes.
• Use Parquet/Delta Lake/Iceberg for storage and lakehouse architectures.
• Build high-performance analytics with Trino or Presto.
• Collaborate with DevOps to manage infra and CI/CD with Azure DevOps.
• Maintain data quality and observability across pipelines.
• Write clean, tested Python and SQL code.
• Mentor and guide a team of data engineers.
• Participate in agile ceremonies and sprint planning.

Skills

Advanced Python

SQL proficiency

Cloud experience (Azure)

Real-time data processing

Spark batch processing

Data orchestration

API development

Team leadership

Agile collaboration

Education

Bachelor's degree in Computer Science or related field

Tools

Kafka

Apache Flink

Spark Structured Streaming

Airflow

Azure Data Factory

Docker

Kubernetes

Helm

Parquet

Delta Lake

Iceberg

Trino

Presto

Terraform

Great Expectations

ClickHouse

Job description

We are seeking a Senior Data Engineer with 6+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.

Key Responsibilities

• Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.
• Develop batch processing workflows with Apache Spark (PySpark)
• Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory
• Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes
• Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg
• Build high-performance analytics engines using tools like Trino or Presto
• Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps
• Ensure data quality and consistency using tools like Great Expectations
• Write modular, well-tested, and maintainable Python and SQL code
• Develop an observability layer to monitor and optimize performance across data pipelines
• Lead and mentor a team of data engineers
• Participate in agile ceremonies and contribute to sprint planning and reviews

Required Skills & Experience

• Advanced Python programming with a strong focus on modular and testable code
• Strong knowledge of SQL and experience working with large-scale datasets
• Hands-on experience with at least one major cloud platform (Azure preferred)
• Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)
• Expertise in Apache Spark (PySpark) for batch processing
• Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)
• Proficient in using Azure Data Factory or Apache Airflow for data orchestration
• Experience in building APIs to expose large datasets
• Solid experience with Docker, Kubernetes, and Helm
• Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg
• Basic experience with Terraform for infrastructure provisioning
• Practical experience with data quality frameworks (e.g., Great Expectations)
• Proven ability in debugging and performance tuning of streaming and batch data jobs
• Experience with AI-driven tools (e.g., text-to-SQL) is a plus
• Bachelor’s degree in computer science or related discipline