Senior Data Engineer: Cloud Pipelines & Airflow

EPAM Systems Inc; Malaysia; On-site; MYR 180,000 - 240,000; Full time

Job summary

EPAM Systems Inc. seeks a Senior Data Engineer to design, build and run reliable ETL/ELT pipelines for enterprise data platforms.

You will use Python, SQL and Apache Airflow across Azure, GCP or AWS, troubleshooting issues and improving automation while partnering with clients on requirements. You will design scalable pipelines, optimize SQL, model data warehouses and orchestrate Airflow workflows to ensure quality, availability and scale across cloud environments.

Qualifications

• Hands-on experience building ETL/ELT pipelines and data warehousing.
• Advanced SQL skills including optimization and data validation.
• Python proficiency for pipeline development and software engineering practices.
• Experience with distributed processing using PySpark or Apache Beam.

Responsibilities

• Design and build scalable ETL and ELT pipelines using Python, SQL and cloud-native services.
• Write, optimize and troubleshoot complex SQL for transformations, validation and performance.
• Create and maintain data models and warehouse structures that support quality, availability and scale.
• Orchestrate workflows in Apache Airflow including scheduling, monitoring and incident resolution.
• Support pipeline delivery and platform operations across Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS).
• Improve continuous integration and continuous delivery (CI/CD), source control and automated deployment for data solutions.
• Investigate data, integration and infrastructure issues, drive root-cause analysis and implement fixes.
• Collaborate with clients and cross-functional teams to align on requirements, risks, dependencies and delivery plans.

Skills

ETL/ELT pipelines

SQL optimization

Python development

Data modeling

CI/CD for data solutions

Stakeholder communication

Streaming concepts

Tools

PySpark

Apache Beam

Apache Airflow

Azure

GCP

AWS

Terraform

Kubernetes

Fivetran

Google Cloud Composer

Kafka

Job description

EPAM Systems Inc. seeks a Senior Data Engineer to design, build and run reliable ETL/ELT pipelines for enterprise data platforms.

You will use Python, SQL and Apache Airflow across Azure, GCP or AWS, troubleshooting issues and improving automation while partnering with clients on requirements. You will design scalable pipelines, optimize SQL, model data warehouses and orchestrate Airflow workflows to ensure quality, availability and scale across cloud environments.