Senior Data Engineer: Real-Time Spark on GCP (Onsite)

Compunnel Inc.; Sunnyvale (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Compunnel, Inc. is seeking an experienced Senior Data Engineer in Sunnyvale, CA to design, build, and optimize real-time data pipelines using Spark, Kafka, and cloud services. The role focuses on scalable data architectures on Google Cloud Platform to support analytics and ML workloads.

The ideal candidate has 8+ years in IT, 4+ years on GCP, and strong Python, SQL, and Spark expertise. Familiarity with data warehouses like BigQuery, Snowflake, and Redshift is desired. Onsite work is required.

Qualifications

• Minimum 8 years of total IT experience.
• Minimum 4 years of recent experience on Google Cloud Platform (GCP).
• Strong Python and SQL; Scala/Java a plus.
• Deep Spark expertise (Spark SQL, DataFrames, Spark Streaming).
• Hands-on streaming tech: Kafka, Flink, or Pub/Sub.
• Experience designing batch and real-time pipelines on GCP.
• Familiar with Snowflake and Redshift data warehouses.
• NoSQL experience and data quality monitoring knowledge.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines with Spark and streaming tech.
• Build scalable data architectures (data lakes, data warehouses, streaming platforms).
• Optimize Spark jobs, SQL queries, and data workflows for performance and cost.
• Implement data quality checks, monitoring, and alerting.
• Support analytics and ML workloads with scalable processing solutions.

Skills

Python

SQL

Spark

PySpark

Airflow

ETL

GCP

Kafka

Flink

Pub/Sub

Tools

BigQuery

Databricks

Snowflake

Redshift

Docker

Kubernetes

Job description

Compunnel, Inc. is seeking an experienced Senior Data Engineer in Sunnyvale, CA to design, build, and optimize real-time data pipelines using Spark, Kafka, and cloud services. The role focuses on scalable data architectures on Google Cloud Platform to support analytics and ML workloads.

The ideal candidate has 8+ years in IT, 4+ years on GCP, and strong Python, SQL, and Spark expertise. Familiarity with data warehouses like BigQuery, Snowflake, and Redshift is desired. Onsite work is required.