Lead Data Engineer — Cloud Pipelines & Airflow

EPAM Systems Inc; Colombia; Remote; COP 120,000,000 - 240,000,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Upskilling & certification courses

LinkedIn Learning library access

Global career opportunities

Job summary

EPAM Systems Inc. seeks a hands-on Lead Data Engineer to build and maintain scalable data pipelines and cloud data solutions on AWS.

The role emphasizes production-grade engineering, automation, and data quality. Responsibilities include creating data pipelines with Python, Pandas, and PySpark, orchestrating workflows with MWAA, crafting efficient queries in Athena, and automating infrastructure with Terraform and GitHub Actions.

Qualifications

• Minimum 5 years of professional data engineering experience.
• At least 1 year leading and managing development teams.
• Strong Python data engineering skills with Pandas.
• Proficient in PySpark for large-scale processing.
• Hands-on experience with AWS MWAA or Apache Airflow to coordinate data workflows.
• Experience with Terraform to automate infrastructure and GitHub Actions for CI/CD.
• Solid knowledge of DevOps practices in data contexts.
• Excellent English communication skills (B1 or higher).

Responsibilities

• Create, construct, and sustain scalable data pipelines on AWS.
• Develop data processing solutions using Python, Pandas, and PySpark.
• Manage MWAA workflows and data orchestration.
• Build and optimize Athena queries for analytics.
• Automate infrastructure setup with Terraform and GitHub Actions.
• Implement testing, monitoring, and data quality safeguards.
• Review code and promote best engineering practices within the team.
• Collaborate with cross-functional teams in Agile setup.

Skills

Python

Pandas

PySpark

AWS

Airflow

DevOps

Tools

Terraform

GitHub Actions

MWAA

Athena

Job description

EPAM Systems Inc. seeks a hands-on Lead Data Engineer to build and maintain scalable data pipelines and cloud data solutions on AWS.

The role emphasizes production-grade engineering, automation, and data quality. Responsibilities include creating data pipelines with Python, Pandas, and PySpark, orchestrating workflows with MWAA, crafting efficient queries in Athena, and automating infrastructure with Terraform and GitHub Actions.