Senior Data Engineer

EPAM Systems; México; Remote; MXN 900.000 - 1.500.000; Vollzeit

Benefits dieser Stelle

Healthcare benefits

Paid time off

Upskilling and certification courses

LinkedIn Learning access

Zusammenfassung

EPAM Systems seeks a Senior Data Engineer to design and deliver scalable cloud-native data solutions on Azure and Databricks. You will craft ETL/ELT pipelines with PySpark, Spark SQL, and Delta Lake, while enforcing data governance and robust quality controls.

You will collaborate with stakeholders to capture requirements, optimize workloads, and drive best practices in CI/CD and infrastructure automation.

Qualifikationen

• 3+ years of experience in Data Engineering or Software Engineering.
• Expertise in Azure Databricks, Apache Spark, and PySpark.
• Proficiency in Spark SQL with cloud-based Azure solutions.
• Advanced Python programming and SQL optimization.
• Knowledge of Unity Catalog and data governance practices.
• Familiarity with Git-based workflows and CI/CD pipelines.
• Experience with Databricks Workflows and Azure Data Factory.
• Strong communication and stakeholder management skills.

Aufgaben

• Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines.
• Deliver cloud-native data solutions using Azure and Databricks technologies.
• Develop ETL/ELT processes with PySpark, Spark SQL, and Delta Lake.
• Apply metadata-driven ingestion frameworks to boost scalability and maintainability.
• Build batch and incremental data processing pipelines and ensure data quality.
• Set up monitoring, logging, and error handling mechanisms; address production issues.
• Track and optimize Databricks workloads for efficiency and cost control.
• Guide team members and shape engineering standards and best practices.

Kenntnisse

Data engineering

Cloud data platforms

Analytical thinking

Problem solving

Stakeholder management

Communication

Python

Ausbildung

Bachelor's degree in a technical field

Tools

Azure Databricks

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Azure Data Factory

Terraform / IaC

Databricks Workflows

Git workflows

CI/CD

Jobbeschreibung

We are seeking a Senior Data Engineer with proven expertise in Azure Data Platform technologies, Databricks, and PySpark to strengthen our growing data engineering organization. In this position, you will craft and deliver scalable, cloud-native data solutions that drive enterprise analytics, reporting, and business intelligence efforts. You will manage large-scale datasets, develop reusable ingestion frameworks, refine data pipelines, and play a key role in evolving modern data platforms built on Azure and Databricks. The right candidate pairs strong hands-on engineering capabilities with a background in designing robust, metadata-driven data solutions, along with a genuine drive to deliver high-quality, maintainable data products.ResponsibilitiesDesign, build, and sustain scalable data ingestion, transformation, and delivery pipelinesDeliver cloud-native data solutions using Azure and Databricks technologiesCreate and refine ETL/ELT processes with PySpark, Spark SQL, and Delta LakeApply metadata-driven and configuration-based ingestion frameworks to boost scalability and maintainabilityBuild and sustain batch and incremental data processing pipelinesSet up data quality controls, validation frameworks, monitoring, logging, and error handling mechanismsWork closely with business stakeholders, architects, and engineering teams to capture requirements and deliver data solutionsAddress production issues and push forward continuous improvements in platform reliability and performanceTrack and fine-tune Databricks workloads for efficiency, scalability, and cost managementUphold engineering best practices including code reviews, automated testing, CI/CD, and infrastructure automationContribute to data governance, security, and access management initiatives using Unity Catalog and Azure servicesGuide team members and shape engineering standards and best practicesRequirements3+ years of experience in Data Engineering or Software EngineeringExpertise in Azure Databricks, Apache Spark, and PySparkProficiency in Spark SQL and a track record of building solutions on Azure cloud platformsAdvanced Python programming skills paired with strong SQL development and optimization experienceBackground in Unity Catalog and enterprise data governance practicesFamiliarity with Git-based source control workflowsExperience designing and maintaining CI/CD pipelinesKnowledge of orchestration tools such as Databricks Workflows and Azure Data Factory (ADF)Strong analytical, troubleshooting, and problem-solving skillsExcellent communication and stakeholder management abilitiesEnglish proficiency at B2 level or higherNice to haveKnowledge of Terraform and Infrastructure as Code (IaC)Experience with Databricks Asset BundlesSkills in building reusable data engineering frameworks and acceleratorsExpertise in advanced Spark performance tuning and optimizationFamiliarity with enterprise-scale lakehouse architecturesExposure to data security, compliance, and governance initiativesExperience working within Agile delivery teamsWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.