Senior Data Engineer: Cloud Data Pipelines & Governance

Tech Mahindra; Kuala Lumpur; On-site; MYR 180,000 - 240,000; Full time

Job summary

Tech Mahindra is seeking a Senior Data Engineer to design, build, and operate scalable data pipelines for both On-Premise and Cloud ecosystems. The role emphasizes Azure Data Lake Storage, Iceberg, and modern data tools across the stack.

You will migrate legacy pipelines, implement robust ETL/ELT processes, and collaborate with cross-functional teams to meet business needs while ensuring data quality and governance.

Qualifications

• Extensive background in building and maintaining data pipelines/ETL processes.
• Experience in migrating legacy data pipelines to modern data stack.
• Hands-on experience with on-premise and cloud ecosystems (Azure preferred).

Responsibilities

• Migrate data pipelines from existing framework to GDP data acquisition framework.
• Configure, develop and deliver data ingestion scripts for loading data into T1 data layer.
• Develop and manage ETL/ELT workflows with data quality checks.
• Integrate and automate data quality checks within pipelines.
• Deploy and manage containerized apps using Docker and orchestrate on Kubernetes.
• Work with data lake/warehouse tech like Iceberg.
• Orchestrate workflows using Airflow.
• Integrate with Datahub and Ranger for governance.
• Collaborate with cross-functional teams to deliver data solutions.
• Ensure security, compliance, and best practices in data management.

Skills

Linux

Python

Shell scripting

Java

Data modelling

Data warehousing

Big data

Docker

Kubernetes

MinIO

Azure Data Lake Storage

S3 protocols

Apache Iceberg

Kafka

Airflow

Datahub

Trino

Ranger

Tools

Docker

Kubernetes

Airflow

Apache Iceberg

Trino

Ranger

Job description

Tech Mahindra is seeking a Senior Data Engineer to design, build, and operate scalable data pipelines for both On-Premise and Cloud ecosystems. The role emphasizes Azure Data Lake Storage, Iceberg, and modern data tools across the stack.

You will migrate legacy pipelines, implement robust ETL/ELT processes, and collaborate with cross-functional teams to meet business needs while ensuring data quality and governance.