Senior Big Data Software Engineer (Databricks + AWS)

SoftServe; Poland; On-site; PLN 180,000 - 260,000; Full time

Job summary

SoftServe is seeking a Data Engineer to contribute to scalable cloud-based data solutions on AWS using Databricks and PySpark. You will build batch and streaming pipelines, optimize data platforms, and collaborate with cross-functional teams to deliver reliable data services across project lifecycles.

You will work with Spark, Delta Lake, and orchestration tools to implement end-to-end data architectures and modernize existing pipelines, enabling efficient data processing and governance.

Qualifications

• 4+ years of experience in Big Data or Data Engineering.
• Proficiency in Python (PySpark) and SQL.
• Hands-on experience with AWS for data engineering.
• Practical Databricks experience on AWS, building and managing pipelines.
• Strong knowledge of Apache Spark and large-scale data processing.
• Experience with batch and streaming data processing.
• Familiarity with Delta Lake and Databricks components like Workflows and Jobs.
• Experience with orchestration tools like Apache Airflow or MWAA.
• Knowledge of streaming technologies (Kafka, MSK, Kinesis).
• Upper-intermediate or higher level of English.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build and optimize data processing solutions using Python (PySpark) and SQL.
• Develop and manage data workflows using Databricks on AWS.
• Work with Apache Spark and Databricks for large-scale data processing.
• Implement and maintain Delta Lake-based data architectures.
• Utilize orchestration tools such as Apache Airflow or MWAA to manage workflows.
• Support implementation of streaming solutions using Apache Kafka, MSK, or Kinesis.
• Collaborate with stakeholders to deliver effective data solutions.
• Participate in architecture discussions and continuous improvement initiatives.
• Participate in full project lifecycle from PoC to production.

Skills

Python (PySpark)

SQL

AWS cloud services

Databricks on AWS

Apache Spark

Delta Lake

Airflow or MWAA

Streaming tech (Kafka/Kinesis)

English communication

Tools

Databricks

Apache Airflow

MWAA

Apache Kafka

Kinesis

Delta Lake

Job description

About The Role

In this role, you will contribute to developing scalable cloud-based data solutions on AWS using Databricks and modern big data technologies. You will work on batch and streaming data pipelines, support optimization and modernization initiatives, and collaborate with cross-functional teams to deliver reliable and efficient data platforms across different stages of the project lifecycle.

About The Role

In this role, you will contribute to developing scalable cloud-based data solutions on AWS using Databricks and modern big data technologies. You will work on batch and streaming data pipelines, support optimization and modernization initiatives, and collaborate with cross-functional teams to deliver reliable and efficient data platforms across different stages of the project lifecycle.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines
• Build and optimize data processing solutions using Python (PySpark) and SQL
• Develop and manage data workflows using Databricks on AWS
• Work with Apache Spark and Databricks for large-scale data processing
• Implement and maintain Delta Lake-based data architectures
• Utilize orchestration tools such as Apache Airflow or MWAA to manage workflows
• Support implementation of streaming solutions using Apache Kafka, Amazon MSK, or Kinesis
• Collaborate with technical and business stakeholders to deliver effective data solutions
• Participate in architecture discussions and contribute to continuous improvement initiatives within the team
• Participate in the full project lifecycle, from PoC and MVP stages to production implementation

Requirements

• 4+ years of experience in Big Data or Data Engineering
• Strong proficiency in Python (PySpark) and SQL
• Hands-on experience with AWS cloud services for data engineering solutions
• Practical experience with Databricks on AWS, including building and managing data pipelines
• Strong knowledge of Apache Spark and large-scale data processing
• Experience with batch and streaming data processing
• Familiarity with Delta Lake and Databricks components such as Workflows and Jobs
• Experience with orchestration tools such as Apache Airflow or MWAA
• Knowledge of streaming technologies such as Apache Kafka, Amazon MSK, or Kinesis
• Upper-intermediate or higher level of English

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.