Senior Big Data Engineer: Spark, Databricks on AWS

SoftServe; Poland; On-site; PLN 180,000 - 260,000; Full time

Job summary

SoftServe is seeking a Data Engineer to contribute to scalable cloud-based data solutions on AWS using Databricks and PySpark. You will build batch and streaming pipelines, optimize data platforms, and collaborate with cross-functional teams to deliver reliable data services across project lifecycles.

You will work with Spark, Delta Lake, and orchestration tools to implement end-to-end data architectures and modernize existing pipelines, enabling efficient data processing and governance.

Qualifications

• 4+ years of experience in Big Data or Data Engineering.
• Proficiency in Python (PySpark) and SQL.
• Hands-on experience with AWS for data engineering.
• Practical Databricks experience on AWS, building and managing pipelines.
• Strong knowledge of Apache Spark and large-scale data processing.
• Experience with batch and streaming data processing.
• Familiarity with Delta Lake and Databricks components like Workflows and Jobs.
• Experience with orchestration tools like Apache Airflow or MWAA.
• Knowledge of streaming technologies (Kafka, MSK, Kinesis).
• Upper-intermediate or higher level of English.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build and optimize data processing solutions using Python (PySpark) and SQL.
• Develop and manage data workflows using Databricks on AWS.
• Work with Apache Spark and Databricks for large-scale data processing.
• Implement and maintain Delta Lake-based data architectures.
• Utilize orchestration tools such as Apache Airflow or MWAA to manage workflows.
• Support implementation of streaming solutions using Apache Kafka, MSK, or Kinesis.
• Collaborate with stakeholders to deliver effective data solutions.
• Participate in architecture discussions and continuous improvement initiatives.
• Participate in full project lifecycle from PoC to production.

Skills

Python (PySpark)

SQL

AWS cloud services

Databricks on AWS

Apache Spark

Delta Lake

Airflow or MWAA

Streaming tech (Kafka/Kinesis)

English communication

Tools

Databricks

Apache Airflow

MWAA

Apache Kafka

Kinesis

Delta Lake

Job description

SoftServe is seeking a Data Engineer to contribute to scalable cloud-based data solutions on AWS using Databricks and PySpark. You will build batch and streaming pipelines, optimize data platforms, and collaborate with cross-functional teams to deliver reliable data services across project lifecycles.

You will work with Spark, Delta Lake, and orchestration tools to implement end-to-end data architectures and modernize existing pipelines, enabling efficient data processing and governance.