Senior Data Engineer: Databricks & PySpark Pipelines

EPAM Systems Inc; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

EPAM Systems is seeking a Senior Data Software Engineer to design and implement scalable data pipelines using PySpark and SQL within a Databricks-centric environment. You will write clean, well-tested code and collaborate with data scientists and engineers to deliver reliable data solutions.

Ideal candidates have 3+ years in data software engineering, strong PySpark/SQL capabilities, experience with Databricks, and a solid grounding in unit testing with pytest; English at B2+ and familiarity

Qualifications

• 3+ years of experience in Data Software Engineering.
• Proven experience with Apache Spark, ideally in a Databricks environment.
• Proficiency in PySpark and SQL.
• Background in unit testing frameworks, especially pytest.
• Understanding of data engineering principles and ETL processes.
• Familiarity with version control systems (e.g., Git).
• Ability to work independently and in a collaborative team setting.
• Excellent problem-solving and communication skills.
• Proficiency in English at an Upper-Intermediate level (B2) or higher.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark (Databricks preferred).
• Write efficient and optimized PySpark code for data transformation and processing.
• Develop and execute complex SQL queries for data extraction, validation, and reporting.
• Implement unit tests using pytest to ensure code reliability and maintainability.
• Collaborate with data scientists, analysts, and other engineers to deliver high-quality data solutions.
• Monitor and troubleshoot data workflows and performance issues.
• Document technical designs, processes, and best practices.

Skills

PySpark

SQL

Unit testing

Apache Spark

Databricks

Tools

Databricks

Git

Delta Lake

MLflow

Job description

EPAM Systems is seeking a Senior Data Software Engineer to design and implement scalable data pipelines using PySpark and SQL within a Databricks-centric environment. You will write clean, well-tested code and collaborate with data scientists and engineers to deliver reliable data solutions.

Ideal candidates have 3+ years in data software engineering, strong PySpark/SQL capabilities, experience with Databricks, and a solid grounding in unit testing with pytest; English at B2+ and familiarity