Senior Cloud Data Engineer (AWS, Spark & Databricks)

SoftServe; Town of Poland (NY); On-site; USD 100,000 - 150,000; Full time

Job summary

SoftServe is seeking a data engineer to design and implement scalable cloud-based data platforms on AWS, using Databricks and PySpark to build batch and streaming pipelines.

You will collaborate with cross-functional teams to optimize data workflows, manage Delta Lake architectures, and apply Airflow or MWAA for orchestration, from PoC to production. Strong English and 4+ years of experience are expected, with hands-on AWS data engineering expertise.

Qualifications

• 4+ years of experience in Big Data or Data Engineering.
• Proficiency in Python (PySpark) and SQL.
• Hands-on experience with AWS data engineering solutions.
• Practical experience with Databricks on AWS.
• Strong knowledge of Apache Spark and large-scale data processing.
• Experience with batch and streaming data processing.
• Familiarity with Delta Lake and Databricks components such as Workflows and Jobs.
• Experience with orchestration tools such as Apache Airflow or MWAA.
• Knowledge of streaming technologies such as Kafka, MSK, or Kinesis.
• Upper-intermediate or higher level of English.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build and optimize data processing solutions using Python (PySpark) and SQL.
• Develop and manage data workflows using Databricks on AWS.
• Work with Apache Spark and Databricks for large-scale data processing.
• Implement and maintain Delta Lake-based data architectures.
• Utilize orchestration tools such as Apache Airflow or MWAA to manage workflows.
• Support implementation of streaming solutions using Apache Kafka, MSK, or Kinesis.
• Collaborate with stakeholders to deliver effective data solutions.
• Participate in architecture discussions and continuous improvement.
• Participate in the full project lifecycle from PoC to production.

Skills

Big Data

Python PySpark

SQL

English Proficiency

Tools

Databricks

Spark

Airflow

MWAA

Delta Lake

Kafka

Kinesis

AWS Cloud

Job description

SoftServe is seeking a data engineer to design and implement scalable cloud-based data platforms on AWS, using Databricks and PySpark to build batch and streaming pipelines.

You will collaborate with cross-functional teams to optimize data workflows, manage Delta Lake architectures, and apply Airflow or MWAA for orchestration, from PoC to production. Strong English and 4+ years of experience are expected, with hands-on AWS data engineering expertise.