Lead Data Engineer: PySpark & AWS Data Lakes

EPAM Systems; Mexico; On-site; PHP 3,685,503 - 5,528,255; Full time

Benefits offered by this job

International projects

Work with global teams

Employee financial programs

Paid time off

Upskilling and certification courses

Access to LinkedIn Learning

Global career opportunities

Volunteer opportunities

Job summary

EPAM Systems is seeking a Lead Data Engineer based in Pampanga, Philippines. This role involves building ETL pipelines and AWS data lake architectures, facilitating integration of structured and unstructured data from various sources including SAP and SQL databases. Candidates should have over 5 years of experience in data engineering, a strong proficiency in Python and PySpark, and hands-on experience with AWS Glue. The position offers international projects, opportunities for upskilling, and a supportive work culture.

Qualifications

• 5+ years of experience in data engineering roles.
• 1+ year of experience managing teams.
• Proficient in Python and PySpark for data pipelines.
• Experience with ETL processes and AWS Glue Jobs.
• Familiarity with Amazon S3 and data cataloging.
• Strong documentation and communication skills.
• English proficiency at B2+ level.

Responsibilities

• Manage ETL pipelines using PySpark and AWS Glue.
• Coordinate workflows with Apache Airflow.
• Integrate data from enterprise systems into AWS.
• Build API interactions for data retrieval.
• Maintain data quality checks and validation.

Skills

Python

PySpark

ETL processes

Apache Airflow

AWS

Data integration

API integration

Data quality checks

Tools

AWS Glue

Amazon S3

Data lake methodologies

Job description

EPAM Systems is seeking a Lead Data Engineer based in Pampanga, Philippines. This role involves building ETL pipelines and AWS data lake architectures, facilitating integration of structured and unstructured data from various sources including SAP and SQL databases. Candidates should have over 5 years of experience in data engineering, a strong proficiency in Python and PySpark, and hands-on experience with AWS Glue. The position offers international projects, opportunities for upskilling, and a supportive work culture.