Lead Data Engineer

EPAM Systems; Argentina; Remote; ARS 123,879,000 - 185,819,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Learning and development opportunities

Global career opportunities

Job summary

EPAM Systems Inc. is seeking a hands-on Lead Data Engineer to build scalable data pipelines on AWS and craft cloud-based data solutions. You will lead development, implement DAGs with Airflow, and drive automation with Terraform and CI/CD via GitHub Actions.

You will apply Python (Pandas) and PySpark to large-scale data processing, coordinate with cross-functional teams in an Agile setup, and ensure robust testing, monitoring, and data quality across production pipelines.

Qualifications

• 5+ years of relevant professional background.
• Experience leading and managing development teams.
• Strong Python for data engineering tasks (Pandas).
• Proficient PySpark for large-scale data processing.
• Experience with AWS MWAA or Airflow to orchestrate workflows.
• Terraform for infrastructure automation and GitHub Actions for CI/CD.

Responsibilities

• Create, construct, and maintain scalable data pipelines on AWS.
• Develop data processing solutions using Python, Pandas, and PySpark.
• Manage and monitor workflows with MWAA/Airflow.
• Automate infrastructure setup and deployments with Terraform and GitHub Actions.
• Implement testing, monitoring, and data-quality safeguards.

Skills

Python

PySpark

AWS

Airflow

DevOps

Automation

Leadership

Tools

Terraform

GitHub Actions

Airflow

MWAA

Athena

Job description

We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practicesResponsibilitiesCreate, construct, and sustain scalable data pipelines running on AWS infrastructureBuild dependable data processing solutions leveraging Python, Pandas, and PySparkEstablish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)Craft and fine-tune data queries through AWS Athena to enable efficient analytical workStreamline infrastructure setup and deployment workflows through automation using Terraform and GitHub ActionsPut in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependableTake part in reviewing code and help promote sound engineering practices throughout the teamWork alongside engineering, product, and business groups within an Agile settingRequirementsAt least 5 years of applicable professional backgroundA minimum of one year spent leading and managing development teamsSolid foundation in core Python programming applied to data engineering workPractical experience applying Python Pandas for manipulating and analyzing dataStrong capability with PySpark for handling large-scale distributed data processingBackground working with AWS MWAA or Apache Airflow to coordinate and manage data workflowsFamiliarity with AWS Athena for running queries and analyzing data housed within AWSHands-on background using Terraform to automate infrastructure setup and managementWorking knowledge of GitHub Actions for constructing automated CI/CD pipelinesGood grasp of DevOps principles as applied within data engineering contextsStrong capabilities in data engineering, troubleshooting, and resolving complex problemsUpper-intermediate English language skills or higher (B1)Nice to haveBackground applying GitHub Copilot to assist with development, code reviews, testing, and release processesKnowledge of CI/CD methodologies and associated tools for smoother software deliveryBackground working within Agile/Scrum team structuresExperience gained within financial services or investment management sectorsBackground in Gen AI application development and incorporating Gen AI practices into the software development lifecycle (SDLC)We offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedIn