Senior Data Engineer

Tech Mahindra; Kuala Lumpur; On-site; MYR 180,000 - 240,000; Full time

Job summary

Tech Mahindra is seeking a Senior Data Engineer to design, build, and operate scalable data pipelines for both On-Premise and Cloud ecosystems. The role emphasizes Azure Data Lake Storage, Iceberg, and modern data tools across the stack.

You will migrate legacy pipelines, implement robust ETL/ELT processes, and collaborate with cross-functional teams to meet business needs while ensuring data quality and governance.

Qualifications

• Extensive background in building and maintaining data pipelines/ETL processes.
• Experience in migrating legacy data pipelines to modern data stack.
• Hands-on experience with on-premise and cloud ecosystems (Azure preferred).

Responsibilities

• Migrate data pipelines from existing framework to GDP data acquisition framework.
• Configure, develop and deliver data ingestion scripts for loading data into T1 data layer.
• Develop and manage ETL/ELT workflows with data quality checks.
• Integrate and automate data quality checks within pipelines.
• Deploy and manage containerized apps using Docker and orchestrate on Kubernetes.
• Work with data lake/warehouse tech like Iceberg.
• Orchestrate workflows using Airflow.
• Integrate with Datahub and Ranger for governance.
• Collaborate with cross-functional teams to deliver data solutions.
• Ensure security, compliance, and best practices in data management.

Skills

Linux

Python

Shell scripting

Java

Data modelling

Data warehousing

Big data

Docker

Kubernetes

MinIO

Azure Data Lake Storage

S3 protocols

Apache Iceberg

Kafka

Airflow

Datahub

Trino

Ranger

Tools

Docker

Kubernetes

Airflow

Apache Iceberg

Trino

Ranger

Job description

Tech Mahindra represents the connected world, offering innovative and customer-centric information technology experiences, enabling Enterprises, Associates, and the Society to Rise. It has 150,000+ professionals working for 1000+ Global Customers (including Fortune 500 companies) in 90 Countries. We’re part of the esteemed Mahindra group, headquartered in India. Under a new CEO, Tech Mahindra is committed to a transformative journey with 'Scale @ Speed' as our guiding principle.

Job Description:

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and platforms On Premise and Cloud Ecosystem (Microsoft Azure Preferrable), with hands-on expertise in modern data engineering tools and frameworks.

Key Responsibilities:

• Migrate data pipelines from existing data acquisition framework to the new GDP data acquisition framework
• Configure, develop and deliver data ingestion scripts for loading data into T1 data layer
• Develop and manage ETL/ELT workflows, ensuring data quality, integrity, and reliability.
• Integrate and automate data quality checks and validation processes within data pipelines.
• Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
• Work with modern data lake and warehouse technologies such as Iceberg
• Orchestrate complex workflows using Airflow.
• Integrate with data catalog and governance tools such as Datahub and Ranger.
• Collaborate with cross-functional teams to understand business requirements and deliver data solutions.
• Ensure security, compliance, and best practices in data management and governance.

Key Skills Required:

• Strong proficiency in Linux, Python, and Shell scripting.
• Hands-on experience with Docker, Kubernetes, and container orchestration.
• Hands-on experience with Minio and Azure Data Lake Storage (ADLS) using S3 protocols.
• Experience with Apache Iceberg, Kafka, Airflow, Datahub, Trino, and Ranger.
• Proficiency in Java for data engineering tasks.
• Solid understanding of data modelling, data warehousing, and big data technologies.

Prior Experience:

• Extensive background in building and maintaining data pipelines/ETL processes.
• Experience in implementing and integrating data quality frameworks.
• Extensive experience in working on migration projects moving from legacy data pipelines to modernized tech stack