Lead Data Engineer – AWS & Databricks (Lakehouse)

SoftServe; Town of Poland (NY); On-site; USD 150,000 - 190,000; Full time

Job summary

SoftServe is seeking a Lead Data Engineer to design and deliver scalable data platforms on AWS with a focus on the Databricks ecosystem. You will guide a team of engineers, shape architectural decisions, and ensure high-quality delivery of both batch and streaming data solutions.

You will collaborate with business and technical stakeholders across the full project lifecycle, from discovery and design to production implementation, within an innovation-driven environment.

Qualifications

• Proven experience as a Lead Data Engineer delivering scalable data platforms and pipelines.

Responsibilities

• Design and build scalable data solutions on AWS using Databricks.
• Lead dataplatform architecture, data modeling, and engineering best practices.
• Mentor data engineering teams and align with stakeholders on delivery roadmaps.
• Drive batch and streaming data processing using Spark, Flink, Kafka, and Kinesis.

Skills

AWS

Databricks

Python

SQL

Spark

Leadership

Stakeholder management

English proficiency

Tools

Databricks Workflows

MWAA

Airflow

Delta Lake

Unity Catalog

Snowflake

Amazon Redshift

Kafka

Kinesis

Job description

SoftServe is seeking a Lead Data Engineer to design and deliver scalable data platforms on AWS with a focus on the Databricks ecosystem. You will guide a team of engineers, shape architectural decisions, and ensure high-quality delivery of both batch and streaming data solutions.

You will collaborate with business and technical stakeholders across the full project lifecycle, from discovery and design to production implementation, within an innovation-driven environment.