Senior/Mid Big Data Engineer - AWS Databricks & Spark

SoftServe; Poland; On-site; PLN 190,000 - 260,000; Full time

Job summary

SoftServe is seeking an experienced Data Engineer to develop scalable cloud-based data solutions on AWS using Databricks and PySpark. You will design batch and streaming pipelines, optimize data workflows, and collaborate with cross-functional teams to deliver reliable data platforms across project lifecycles.

Responsibilities include building data pipelines, managing Delta Lake architectures, and implementing streaming solutions with Kafka, MSK, or Kinesis, while ensuring high performance and

Qualifications

• 4+ years of experience in Big Data or Data Engineering.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build and optimize data processing solutions using Python (PySpark) and SQL.
• Develop and manage data workflows using Databricks on AWS.
• Work with Apache Spark and Databricks for large-scale data processing.
• Implement and maintain Delta Lake-based data architectures.
• Utilize orchestration tools such as Apache Airflow or MWAA to manage workflows.
• Support implementation of streaming solutions using Apache Kafka, Amazon MSK, or Kinesis.
• Collaborate with technical and business stakeholders to deliver effective data solutions.
• Participate in architecture discussions and continuous improvement initiatives.
• Participate in the full project lifecycle from PoC to production.

Skills

Python (PySpark)

SQL

AWS

Databricks on AWS

Apache Spark

Airflow/MWAA

Kafka/Kinesis

Delta Lake

English (B2+)

Tools

Databricks

Airflow

MWAA

Apache Kafka

Kinesis

Job description

SoftServe is seeking an experienced Data Engineer to develop scalable cloud-based data solutions on AWS using Databricks and PySpark. You will design batch and streaming pipelines, optimize data workflows, and collaborate with cross-functional teams to deliver reliable data platforms across project lifecycles.

Responsibilities include building data pipelines, managing Delta Lake architectures, and implementing streaming solutions with Kafka, MSK, or Kinesis, while ensuring high performance and