Senior Data Engineer/ Databricks, PySpark, Python

EPAM Systems, Inc.; Turkey; On-site; TRY 5,395,000 - 8,337,000; Full time

Benefits offered by this job

Continuous upskilling

Private health insurance

English courses

Well-being events

Job summary

EPAM Systems, Inc. is seeking a Senior Data Engineer / Databricks Engineer to join a project focused on collecting, processing, and analyzing vehicle telemetry data for a major European automotive manufacturer.

The platform operates on Azure Databricks, handling tens of terabytes and ~200 Databricks jobs across Europe, North America and China. You will monitor health, optimize PySpark workloads, support the existing pipelines, and design scalable, cost-aware solutions with strong observability

Qualifications

• 3+ years of hands-on data engineering experience.
• Experience with Azure Databricks and PySpark for large-scale processing.
• Strong Python development skills and performance optimization knowledge.

Responsibilities

• Monitor health, stability, and performance of production data platform.
• Investigate and resolve performance issues and incidents.
• Analyze workloads and identify bottlenecks; optimize PySpark and Databricks jobs.
• Support and improve hundreds of Databricks jobs and data pipelines.
• Prepare platform for increasing data volumes and onboarded platforms.
• Design scaling strategies with focus on predictable infrastructure and costs.
• Track and optimize resource consumption; improve observability and alerts.
• Ensure consistency across multi-region deployments (EU, NA, CN).
• Identify opportunities for automation and cost reduction.

Skills

Azure Databricks

PySpark

Python

Distributed data processing

Performance optimization

Monitoring/Observability

CI/CD (GitHub Actions)

Cloud cost optimization

English (B2+)

Education

Bachelor's degree in Computer Science

Tools

GitHub Actions

Azure

Job description

We are looking for an experienced Senior Data Engineer / Databricks Engineer to join a project focused on collecting, processing, and analyzing vehicle telemetry data for a major European automotive manufacturer.The platform operates at a significant scale, processing tens of terabytes of telemetry data in Azure Databricks and running approximately 200 Databricks jobs. The solution is currently in the production maintenance and evolution phase and is deployed across multiple regions, including Europe, North America, and China, with an additional deployment in China planned in the near future.A key focus of the role will be ensuring that the system can scale reliably while keeping infrastructure and operational costs predictable.ResponsibilitiesMonitor the health, stability, and performance of the production data platformInvestigate and resolve performance issues and production incidentsAnalyze Databricks workloads and identify performance bottlenecksOptimize PySpark workloads and Databricks jobs for performance, reliability, and cost efficiencySupport and improve approximately 200 existing Databricks jobs and associated data pipelinesPrepare the platform for increasing data volumes and workloads as additional vehicle platforms are onboardedDesign and implement scaling strategies with a strong focus on predictable infrastructure and operational costsTrack and optimize Databricks and Azure resource consumptionImprove platform observability, monitoring, alerting, and operational processesEnsure consistency across multiple regional deployments in Europe, North America, and ChinaIdentify opportunities for technical improvements, automation, and reduction of operational overheadRequirements3+ years of hands-on experience in data engineeringExpertise in Azure DatabricksProficiency in PySpark and distributed data processingAdvanced skills in Python developmentUnderstanding of performance optimization for large-scale Spark workloadsBackground in operating and troubleshooting production data platformsKnowledge of monitoring, observability, and capacity planning combined with cloud cost optimizationAbility to analyze existing systems, identify bottlenecks, and propose pragmatic improvementsEnglish proficiency at an Upper-Intermediate level (B2) or higherNice to haveFamiliarity with Microsoft Azure services and cloud infrastructureExperience building and maintaining CI/CD pipelines using GitHub ActionsExposure to large-scale telemetry, IoT, or automotive dataBackground in managing Databricks environments with a large number of scheduled jobs and pipelinesShowcase of multi-region or geographically distributed cloud deploymentsWe offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.