Senior Data Engineer (Cloud)

EPAM Systems; Kuala Lumpur; On-site; MYR 120,000 - 180,000; Full time

Job summary

EPAM Systems, Inc. is seeking a Senior Data Engineer to design, build and run reliable ETL and ELT pipelines for enterprise data platforms, using Python, SQL and Apache Airflow across Microsoft Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS).

You will design data models, optimize transformations and ensure scalability for analytics at scale in Kuala Lumpur, Malaysia. Work with cross-functional teams and clients to translate requirements into robust data solutions, troubleshoot

Qualifications

• ETL/ELT pipelines and data warehousing experience.
• Advanced SQL for transformations, validation and performance.
• Python for pipeline development and software engineering practices.
• Cloud experience across Azure, GCP or AWS.
• Experience with PySpark or Apache Beam.
• Airflow or Cloud Composer for workflow orchestration.

Responsibilities

• Design and build scalable ETL and ELT pipelines.
• Write, optimize and troubleshoot complex SQL for transformations.
• Create and maintain data models and warehouse structures.
• Orchestrate workflows in Apache Airflow.
• Support CI/CD and automated deployment for data solutions.
• Collaborate with clients and cross-functional teams on requirements.

Skills

Python

SQL

Airflow

Cloud platforms

PySpark/Beam

CI/CD

Data modeling

Kubernetes

Terraform

Communication

Tools

Apache Airflow

PySpark

Terraform

Kubernetes

Fivetran

Job description

We are seeking a Senior Data Engineer to design, build and run reliable ETL and ELT pipelines for enterprise data platforms. You will use Python, SQL and Apache Airflow with cloud services across Microsoft Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS). You will troubleshoot issues, improve delivery automation and partner with clients on requirements and trade-offs.ResponsibilitiesDesign and build scalable ETL and ELT pipelines using Python, SQL and cloud-native servicesWrite, optimize and troubleshoot complex SQL for transformations, validation and performanceCreate and maintain data models and warehouse structures that support quality, availability and scaleOrchestrate workflows in Apache Airflow including scheduling, monitoring and incident resolutionSupport pipeline delivery and platform operations across Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS)Improve continuous integration and continuous delivery (CI/CD), source control and automated deployment for data solutionsInvestigate data, integration and infrastructure issues, drive root-cause analysis and implement fixesCollaborate with clients and cross-functional teams to align on requirements, risks, dependencies and delivery plansRequirementsHands-on experience building ETL and ELT pipelines and supporting data warehousingAdvanced SQL skill sets including optimization, data validation and complex transformationsPython proficiency for pipeline development and software engineering practicesExperience with distributed processing using PySpark or Apache BeamPractical cloud experience with Microsoft Azure, Google Cloud Platform (GCP) or Amazon Web Services (AWS) for data workloadWorkflow orchestration background with Apache Airflow or Google Cloud ComposerKnowledge of data modeling, data lifecycle management and data quality controlClear communication with the ability to explain trade-offs, risks and solution options to stakeholdersNice to haveInfrastructure as code exposure using Terraform or KubernetesStreaming familiarity with Kafka and event-driven patternsData integration tooling experience with FivetranWe offerBy choosing EPAM, you're getting a job at one of the most loved workplaces according to Newsweek 2021 & 2022&2023.Employee ideas are the main driver of our business. We have a very supportive environment where your voice mattersYou will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practicesWe offer a transparent career path and an individual roadmap to engineer your future & accelerate your journeyAt EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with usLife at EPAMEPAM is a leader in the fastest-growing segment (product development/digital platform engineering) of the IT industry. We acquired Just-BI in 2021 to reinforce our leading position as a global Business Intelligence services provider and have been growing rapidly. With a talented multinational team, we provide data and analytics expertiseWe are currently involved in end-to-end BI design and implementation projects in major national and international companies. We are proud of our entrepreneurial start-up culture and are focused on investing in people by creating continuous learning and development opportunities for our employees who deliver engineering excellence for our clients