Senior Big Data Engineer — Python & GCP Mastery

SoftServe; Town of Poland (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

SoftServe is seeking a senior data engineer to design and build scalable data pipelines on Google Cloud Platform. You will work with batch and streaming workloads, leveraging Spark, Databricks, and modern data architectures to unlock data value for clients.

You will collaborate with data scientists, architects, and engineers to translate business needs into robust data solutions, ensure data quality, and support end-to-end deployments on GCP services such as BigQuery, Dataflow, Pub/Sub, and

Qualifications

• 5+ years of professional experience in Big Data or Data Engineering.

Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and streaming workloads
• Build and optimize data processing solutions using Python (must), SQL, Java, Apache Spark, and Databricks
• Create cloud-native data architectures leveraging GCP services including BigQuery, Dataflow, Cloud Composer, Pub/Sub, and Cloud Storage
• Implement data transformation, modeling, and analytics engineering practices using dbt and Dataform
• Collaborate with business stakeholders, architects, and engineering teams to translate data requirements into effective technical solutions
• Support development of modern data platforms and analytics solutions on GCP with GCP native services and/or Databricks
• Contribute to technical design discussions, architecture decisions and continuous improvement initiatives
• Ensure data quality, reliability and performance across data processing workflows
• Support end-to-end project delivery, including PoCs, MVPs, production deployments and platform enhancements

Skills

Python

SQL

Java

Apache Spark

Databricks

GCP (Google Cloud Platform)

Airflow / Cloud Composer

Kafka / Pub/Sub

BigQuery

dbt

Dataform

Data pipelines

Tools

Databricks

Apache Airflow

Cloud Composer

BigQuery

Dataflow

Pub/Sub

Apache Beam

Kafka

dbt

Dataform

Spark

Job description

SoftServe is seeking a senior data engineer to design and build scalable data pipelines on Google Cloud Platform. You will work with batch and streaming workloads, leveraging Spark, Databricks, and modern data architectures to unlock data value for clients.

You will collaborate with data scientists, architects, and engineers to translate business needs into robust data solutions, ensure data quality, and support end-to-end deployments on GCP services such as BigQuery, Dataflow, Pub/Sub, and