Lead Data Engineer

EPAM Systems Inc; México; Remote; MXN 420,000 - 650,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Upskilling and certification courses

LinkedIn Learning access

Global career opportunities

Job summary

EPAM Systems Inc. is seeking a hands-on Lead Data Engineer to design and maintain scalable data pipelines on AWS.

You will leverage Python, Pandas, and PySpark to build robust data solutions and orchestrate workflows using MWAA, while driving automation with Terraform and GitHub Actions. You will work within an Agile environment, review code, promote best practices, and collaborate with cross-functional teams across engineering, product, and business units.

Qualifications

• 5+ years of professional experience in data engineering or related field.
• Experience leading and managing development teams.
• Strong Python programming skills for data engineering tasks.
• Hands-on PySpark experience for large-scale data processing.
• Experience with AWS MWAA or Apache Airflow to coordinate data workflows.
• Terraform automation of infrastructure and CI/CD pipelines with GitHub Actions.
• DevOps practices applied to data engineering environments.
• Knowledge of data quality, testing, and monitoring practices.

Responsibilities

• Build scalable data pipelines on AWS infrastructure.
• Develop data processing solutions using Python, Pandas, and PySpark.
• Coordinate and manage workflows with AWS MWAA.
• Query data efficiently using AWS Athena when applicable.
• Automate infrastructure with Terraform and GitHub Actions.
• Contribute to testing, monitoring, and data quality safeguards.
• Review code and promote engineering best practices within the team.
• Collaborate with engineering, product, and business in an Agile setting.

Skills

Python

PySpark

AWS

Airflow

Pandas

Terraform

GitHub Actions

CI/CD

DevOps

Data engineering

Agile/Scrum

Gen AI

GitHub Copilot

Tools

MWAA

Job description

We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practicesResponsibilitiesCreate, construct, and sustain scalable data pipelines running on AWS infrastructureBuild dependable data processing solutions leveraging Python, Pandas, and PySparkEstablish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)Craft and fine-tune data queries through AWS Athena to enable efficient analytical workStreamline infrastructure setup and deployment workflows through automation using Terraform and GitHub ActionsPut in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependableTake part in reviewing code and help promote sound engineering practices throughout the teamWork alongside engineering, product, and business groups within an Agile settingRequirementsAt least 5 years of applicable professional backgroundA minimum of one year spent leading and managing development teamsSolid foundation in core Python programming applied to data engineering workPractical experience applying Python Pandas for manipulating and analyzing dataStrong capability with PySpark for handling large-scale distributed data processingBackground working with AWS MWAA or Apache Airflow to coordinate and manage data workflowsFamiliarity with AWS Athena for running queries and analyzing data housed within AWSHands-on background using Terraform to automate infrastructure setup and managementWorking knowledge of GitHub Actions for constructing automated CI/CD pipelinesGood grasp of DevOps principles as applied within data engineering contextsStrong capabilities in data engineering, troubleshooting, and resolving complex problemsUpper-intermediate English language skills or higher (B1)Nice to haveBackground applying GitHub Copilot to assist with development, code reviews, testing, and release processesKnowledge of CI/CD methodologies and associated tools for smoother software deliveryBackground working within Agile/Scrum team structuresExperience gained within financial services or investment management sectorsBackground in Gen AI application development and incorporating Gen AI practices into the software development lifecycle (SDLC)We offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.