Lead Data Engineer (GCP)

EPAM Systems; Kuala Lumpur; On-site; MYR 90,000 - 150,000; Full time

Benefits offered by this job

Growth opportunities

Global team

Mentoring programs

Certification support

Online courses

Job summary

EPAM Systems is seeking a Lead Data Engineer in Kuala Lumpur to build scalable data pipelines using Python, SQL, Airflow, and BigQuery on Google Cloud Platform. You will contribute to governance, performance, and platform standards while collaborating across engineering, analytics and product teams.

You will design, implement, and monitor ETL/ELT workflows, craft data models for warehouses, and help advance data infrastructure on GCP with best practices.

Qualifications

• Hands-on Python for data transformation and pipelines.
• Experience with Apache Airflow for orchestration.
• GCP including BigQuery and data warehousing concepts.
• Strong data modeling and ETL/ELT design understanding.
• CI/CD practices and GitHub collaboration.

Responsibilities

• Design, build, and optimize scalable data pipelines in Python/SQL.
• Develop and operate Airflow-based ETL workflows.
• Create and tune data models and warehouse structures in BigQuery.
• Support data infrastructure on GCP and governance standards.
• Collaborate with engineering, analytics and product teams.
• Document designs and runbooks in Jira/Confluence.
• Improve governance, architecture, and platform performance.

Skills

Python data transformation

Airflow orchestration

GCP & BigQuery

Data modeling & ETL

CI/CD & GitHub

Cross-team collaboration

Tools

Airflow

BigQuery

GCP

Job description

We are seeking a Lead Data Engineer to build scalable pipelines across multiple data domains. You will use Python, SQL, Apache Airflow and BigQuery to deliver reliable cloud data products on Google Cloud Platform (GCP). You will also help strengthen governance, performance and platform standards.

Responsibilities

• Design, build, optimize and monitor scalable data pipelines using Python and SQL
• Develop and operate ETL/ELT workflows orchestrated with Apache Airflow
• Create, maintain and tune data models and warehouse structures in BigQuery
• Build and support data infrastructure and platform operations on Google Cloud Platform (GCP)
• Partner with engineering, analytics and product teams to deliver trusted datasets and data products
• Document designs, changes and runbooks in Jira and Confluence
• Improve data governance, architectural standards and platform performance

Requirements

• Hands-on experience with Python for data transformation and pipeline development
• Practical knowledge of Apache Airflow for orchestration and scheduling
• Experience with Google Cloud Platform (GCP), including BigQuery
• Strong understanding of data modeling, ETL design and data warehousing concepts
• Familiarity with continuous integration and continuous delivery (CI/CD) practices and GitHubClear communication and confidence collaborating across teams

Nice to have

• Experience with Google Cloud Dataflow
• Exposure to Business Intelligence Markup Language (BIML) for metadata-driven ETL
• Familiarity with infrastructure as code such as Terraform

We offer

• By choosing EPAM, you're getting a job at one of the most loved workplaces according to Newsweek 2021 & 2022&2023.
• Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters
• You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices
• We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey
• At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us