Lead Data Engineer

EPAM Systems Inc; Brasil; Remote; BRL 250,000 - 420,000; Full time

Benefits offered by this job

Healthcare benefits

Learning & development programs

Global career opportunities

Job summary

EPAM Systems Inc is seeking a hands-on Lead Data Engineer to build scalable data pipelines and cloud-based data solutions on AWS. You will leverage Python, Pandas, and PySpark to process large datasets, orchestrate workflows via MWAA, and query data with Athena.

Automation with Terraform and GitHub Actions will streamline deployments and CI/CD pipelines. You will lead development efforts in an Agile environment, review code, and promote sound engineering practices across cross-functional teams,

Qualifications

• Five+ years of professional experience in data engineering.
• Experience leading development teams (minimum 1 year).
• Strong Python data engineering background with Pandas.
• Hands-on PySpark for large-scale processing.
• Experience with AWS MWAA/Airflow for data orchestration.

Responsibilities

• Build scalable data pipelines on AWS infrastructure.
• Develop data solutions using Python, Pandas and PySpark.
• Orchestrate workflows with MWAA/Airflow; coordinate via AWS services like Athena.
• Automate infrastructure with Terraform and GitHub Actions.
• Implement testing, monitoring, and data quality safeguards.
• Review code and promote engineering best practices in an Agile setting.

Skills

Python

Pandas

PySpark

DevOps

English (B1+)

Tools

AWS

Airflow/MWAA

Terraform

GitHub Actions

AWS Athena

Pandas

Job description

We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practicesResponsibilitiesCreate, construct, and sustain scalable data pipelines running on AWS infrastructureBuild dependable data processing solutions leveraging Python, Pandas, and PySparkEstablish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)Craft and fine-tune data queries through AWS Athena to enable efficient analytical workStreamline infrastructure setup and deployment workflows through automation using Terraform and GitHub ActionsPut in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependableTake part in reviewing code and help promote sound engineering practices throughout the teamWork alongside engineering, product, and business groups within an Agile settingRequirementsAt least 5 years of applicable professional backgroundA minimum of one year spent leading and managing development teamsSolid foundation in core Python programming applied to data engineering workPractical experience applying Python Pandas for manipulating and analyzing dataStrong capability with PySpark for handling large-scale distributed data processingBackground working with AWS MWAA or Apache Airflow to coordinate and manage data workflowsFamiliarity with AWS Athena for running queries and analyzing data housed within AWSHands-on background using Terraform to automate infrastructure setup and managementWorking knowledge of GitHub Actions for constructing automated CI/CD pipelinesGood grasp of DevOps principles as applied within data engineering contextsStrong capabilities in data engineering, troubleshooting, and resolving complex problemsUpper-intermediate English language skills or higher (B1)Nice to haveBackground applying GitHub Copilot to assist with development, code reviews, testing, and release processesKnowledge of CI/CD methodologies and associated tools for smoother software deliveryBackground working within Agile/Scrum team structuresExperience gained within financial services or investment management sectorsBackground in Gen AI application development and incorporating Gen AI practices into the software development lifecycle (SDLC)We offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.