Tech Lead, Big Data on GCP & Databricks

SoftServe; Town of Poland (NY); On-site; USD 120,000 - 190,000; Full time

Job summary

SoftServe is seeking a senior data engineering leader to design and implement scalable data platforms on Google Cloud Platform and Databricks. You will collaborate with stakeholders from discovery to production, translating requirements into robust, scalable analytics solutions.

The role focuses on Python/Scala, SQL, Spark (PySpark), and cloud data services, building batch and streaming pipelines, ensuring data quality, governance, and performance, and guiding engineering teams with best

Qualifications

• Proven experience as a Tech Lead, Senior Data Engineer, or Associate Data Architect designing enterprise-scale data platforms
• Strong proficiency in Python or Scala, combined with advanced SQL skills
• Extensive hands-on experience with Apache Spark (PySpark/Spark SQL)
• Experience with Databricks or Google Cloud data services such as BigQuery, Dataflow, Dataproc, or Cloud Composer
• Strong understanding of modern data modeling, data warehousing, and pipeline orchestration using technologies such as Airflow or Databricks Workflows
• Strong communication and stakeholder management skills with the ability to explain technical concepts to both technical and business audiences
• Upper-intermediate or higher level of English
• Experience with Databricks governance capabilities (e.g., Unity Catalog, Delta Lake, or Delta Live Tables), cloud migration or multi-cloud data platforms, pre-sales activities, or industry certifications (Databricks or Google Cloud) is a plus

Responsibilities

• Design and support implementation of scalable, high-performance data platforms using Databricks and Google Cloud data services
• Build and optimize data processing solutions using Python, SQL, Apache Spark, and modern cloud-native technologies
• Design and implement batch and streaming data pipelines, data models, and analytics solutions
• Collaborate with technical and business stakeholders to translate business requirements into technical solutions and implementation plans
• Guide engineering teams by promoting technical best practices and reviewing solution designs
• Participate in architecture discussions, technical evaluations, and Proof of Concept (PoC) activities
• Ensure data quality, reliability, governance, and performance across cloud-based data platforms
• Contribute to continuous improvement initiatives and knowledge sharing within the engineering community

Skills

Python

Scala

SQL

PySpark

Spark SQL

Databricks

BigQuery

Dataflow

Dataproc

Cloud Composer

Airflow

Unity Catalog

Delta Lake

Delta Live Tables

Tools

Databricks

BigQuery

Dataflow

Dataproc

Cloud Composer

Airflow

Unity Catalog

Delta Lake

Delta Live Tables

Job description

SoftServe is seeking a senior data engineering leader to design and implement scalable data platforms on Google Cloud Platform and Databricks. You will collaborate with stakeholders from discovery to production, translating requirements into robust, scalable analytics solutions.

The role focuses on Python/Scala, SQL, Spark (PySpark), and cloud data services, building batch and streaming pipelines, ensuring data quality, governance, and performance, and guiding engineering teams with best