Senior Data Engineer - Databricks/PySpark on Azure

EPAM Systems Inc; United States; Remote; USD 130,000 - 180,000; Full time

Job summary

EPAM Systems Inc. seeks a Senior Data Software Engineer to join the Data Engineering CoE, building scalable data pipelines in Databricks with PySpark and Python to deliver data products for stakeholders.

You will support a team of data engineers, develop and optimize Databricks pipelines, and work with ADLS Gen2 and SAP HANA Data Lake in Delta/Parquet format, implementing data quality checks and collaborating with stakeholders. Strong Azure experience and English at B2+ are required.

Qualifications

• 3+ years of experience in data engineering or a related field.
• Proficiency in Python, PySpark, and Databricks, including Delta Lake.
• Familiarity with GitHub and Git; 경험 with CI/CD via GitHub Actions.
• Knowledge of data lake technologies and MS Azure environments.
• English proficiency at B2 level or higher.
• Nice to have familiarity with Java, SQL, or Scala.

Responsibilities

• Support a team of data engineers to build pipelines used by MLOps and ML Engineers on modeling teams.
• Develop, optimize, and maintain data transformation pipelines in Databricks (PySpark).
• Work with data stored in ADLS Gen2 and SAP HANA Data Lake, primarily in Delta/Parquet format.
• Implement and maintain data quality checks, including schema validation, deduplication, enrichment, and tagging.
• Communicate with stakeholders to understand business processes and model input data.
• Tune performance for large-scale datasets.

Skills

Python

PySpark

Databricks

Delta Lake

GitHub

Git

CI/CD

Azure

English (B2+)

Java

SQL

Scala

Tools

Databricks

Delta Lake

GitHub Actions

Git

Azure

ADLS Gen2

SAP HANA Data Lake

Job description

EPAM Systems Inc. seeks a Senior Data Software Engineer to join the Data Engineering CoE, building scalable data pipelines in Databricks with PySpark and Python to deliver data products for stakeholders.

You will support a team of data engineers, develop and optimize Databricks pipelines, and work with ADLS Gen2 and SAP HANA Data Lake in Delta/Parquet format, implementing data quality checks and collaborating with stakeholders. Strong Azure experience and English at B2+ are required.