Lead Data Engineer

EPAM Systems Inc; Colombia; Remote; COP 120,000,000 - 240,000,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Upskilling & certification courses

LinkedIn Learning library access

Global career opportunities

Job summary

EPAM Systems Inc. seeks a hands-on Lead Data Engineer to build and maintain scalable data pipelines and cloud data solutions on AWS.

The role emphasizes production-grade engineering, automation, and data quality. Responsibilities include creating data pipelines with Python, Pandas, and PySpark, orchestrating workflows with MWAA, crafting efficient queries in Athena, and automating infrastructure with Terraform and GitHub Actions.

Qualifications

• Minimum 5 years of professional data engineering experience.
• At least 1 year leading and managing development teams.
• Strong Python data engineering skills with Pandas.
• Proficient in PySpark for large-scale processing.
• Hands-on experience with AWS MWAA or Apache Airflow to coordinate data workflows.
• Experience with Terraform to automate infrastructure and GitHub Actions for CI/CD.
• Solid knowledge of DevOps practices in data contexts.
• Excellent English communication skills (B1 or higher).

Responsibilities

• Create, construct, and sustain scalable data pipelines on AWS.
• Develop data processing solutions using Python, Pandas, and PySpark.
• Manage MWAA workflows and data orchestration.
• Build and optimize Athena queries for analytics.
• Automate infrastructure setup with Terraform and GitHub Actions.
• Implement testing, monitoring, and data quality safeguards.
• Review code and promote best engineering practices within the team.
• Collaborate with cross-functional teams in Agile setup.

Skills

Python

Pandas

PySpark

AWS

Airflow

DevOps

Tools

Terraform

GitHub Actions

MWAA

Athena

Job description

We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practicesResponsibilitiesCreate, construct, and sustain scalable data pipelines running on AWS infrastructureBuild dependable data processing solutions leveraging Python, Pandas, and PySparkEstablish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)Craft and fine-tune data queries through AWS Athena to enable efficient analytical workStreamline infrastructure setup and deployment workflows through automation using Terraform and GitHub ActionsPut in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependableTake part in reviewing code and help promote sound engineering practices throughout the teamWork alongside engineering, product, and business groups within an Agile settingRequirementsAt least 5 years of applicable professional backgroundA minimum of one year spent leading and managing development teamsSolid foundation in core Python programming applied to data engineering workPractical experience applying Python Pandas for manipulating and analyzing dataStrong capability with PySpark for handling large-scale distributed data processingBackground working with AWS MWAA or Apache Airflow to coordinate and manage data workflowsFamiliarity with AWS Athena for running queries and analyzing data housed within AWSHands-on background using Terraform to automate infrastructure setup and managementWorking knowledge of GitHub Actions for constructing automated CI/CD pipelinesGood grasp of DevOps principles as applied within data engineering contextsStrong capabilities in data engineering, troubleshooting, and resolving complex problemsUpper-intermediate English language skills or higher (B1)Nice to haveBackground applying GitHub Copilot to assist with development, code reviews, testing, and release processesKnowledge of CI/CD methodologies and associated tools for smoother software deliveryBackground working within Agile/Scrum team structuresExperience gained within financial services or investment management sectorsBackground in Gen AI application development and incorporating Gen AI practices into the software development lifecycle (SDLC)We offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.