Lead BigData Engineer (Databricks + AWS)

SoftServe; Town of Poland (NY); On-site; USD 150,000 - 190,000; Full time

Job summary

SoftServe is seeking a Lead Data Engineer to design and deliver scalable data platforms on AWS with a focus on the Databricks ecosystem. You will guide a team of engineers, shape architectural decisions, and ensure high-quality delivery of both batch and streaming data solutions.

You will collaborate with business and technical stakeholders across the full project lifecycle, from discovery and design to production implementation, within an innovation-driven environment.

Qualifications

• Proven experience as a Lead Data Engineer delivering scalable data platforms and pipelines.

Responsibilities

• Design and build scalable data solutions on AWS using Databricks.
• Lead dataplatform architecture, data modeling, and engineering best practices.
• Mentor data engineering teams and align with stakeholders on delivery roadmaps.
• Drive batch and streaming data processing using Spark, Flink, Kafka, and Kinesis.

Skills

AWS

Databricks

Python

SQL

Spark

Leadership

Stakeholder management

English proficiency

Tools

Databricks Workflows

MWAA

Airflow

Delta Lake

Unity Catalog

Snowflake

Amazon Redshift

Kafka

Kinesis

Job description

About the Role

In this role, you will lead the design and development of scalable data platforms on AWS, with a strong focus on the Databricks ecosystem. You will guide a team of engineers, shape architectural decisions, and ensure high-quality delivery of both batch and streaming data solutions. You will work closely with business and technical stakeholders, contributing across the full project lifecycle, from discovery and design to production implementation, within a collaborative and innovation-driven environment.

Responsibilities

• Design, build, and optimize scalable data solutions on AWS using Databricks, including Lakehouse architectures based on Delta Lake and Unity Catalog
• Develop and enhance batch and real-time data processing solutions using technologies such as Apache Spark, Flink, Kafka, Amazon MSK, and Kinesis
• Lead data integration and migration activities, including source-to-target mapping, data ingestion, transformation, and quality assurance across multiple data sources
• Drive data platform architecture, data modeling, and engineering best practices to ensure scalability, reliability, and long-term maintainability
• Collaborate with business and technical stakeholders to translate requirements into effective data solutions, implementation roadmaps, and delivery plans
• Support and mentor data engineering teams, helping to define priorities, promote technical excellence, and enable successful project delivery
• Build and manage automated workflows and data pipelines using orchestration and analytics technologies such as Databricks Workflows, Apache Airflow, MWAA, Snowflake, and Amazon Redshift
• Contribute across the full solution lifecycle to explore emerging technologies and share knowledge within the engineering community

Requirements

• Proven experience as a Lead Data Engineer, with a strong background in designing and delivering scalable data platforms and pipelines
• Hands‑on expertise in batch and real‑time data processing using technologies such such as Apache Spark, Flink, Kafka, Amazon MSK, or Kinesis
• Strong experience with AWS and Databricks, including Delta Lake, Unity Catalog, Workflows, and Jobs
• Proficiency in Python (preferred), Scala, or Java, combined with advanced SQL skills
• Experience building and orchestrating data workflows using tools such as Databricks Workflows, Apache Airflow, or MWAA
• Practical knowledge of modern data warehousing and analytics solutions, including Snowflake and Amazon Redshift
• Familiarity with data engineering best practices, version control, and data formats, including GitHub, Avro, and SQL‑based systems
• Strong leadership, stakeholder management, and communication skills, with the ability to translate business requirements into technical solutions and guide teams to successful delivery
• Upper‑intermediate or higher level of English

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.