Senior Data Engineer

Compunnel Inc.; Sunnyvale (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Compunnel, Inc. is seeking an experienced Senior Data Engineer in Sunnyvale, CA to design, build, and optimize real-time data pipelines using Spark, Kafka, and cloud services. The role focuses on scalable data architectures on Google Cloud Platform to support analytics and ML workloads.

The ideal candidate has 8+ years in IT, 4+ years on GCP, and strong Python, SQL, and Spark expertise. Familiarity with data warehouses like BigQuery, Snowflake, and Redshift is desired. Onsite work is required.

Qualifications

• Minimum 8 years of total IT experience.
• Minimum 4 years of recent experience on Google Cloud Platform (GCP).
• Strong Python and SQL; Scala/Java a plus.
• Deep Spark expertise (Spark SQL, DataFrames, Spark Streaming).
• Hands-on streaming tech: Kafka, Flink, or Pub/Sub.
• Experience designing batch and real-time pipelines on GCP.
• Familiar with Snowflake and Redshift data warehouses.
• NoSQL experience and data quality monitoring knowledge.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines with Spark and streaming tech.
• Build scalable data architectures (data lakes, data warehouses, streaming platforms).
• Optimize Spark jobs, SQL queries, and data workflows for performance and cost.
• Implement data quality checks, monitoring, and alerting.
• Support analytics and ML workloads with scalable processing solutions.

Skills

Python

SQL

Spark

PySpark

Airflow

ETL

GCP

Kafka

Flink

Pub/Sub

Tools

BigQuery

Databricks

Snowflake

Redshift

Docker

Kubernetes

Job description

Job Summary

We are seeking an experienced Senior Data Engineer with strong Spark and streaming expertise to build real-time, scalable data pipelines using technologies such as Spark, Kafka, and cloud services. The ideal candidate will have experience designing and optimizing data solutions on GCP to support analytics and machine learning workloads.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies such as Kafka and Flink.
• Build and optimize scalable data architectures, including data lakes, data warehouses such as BigQuery, and streaming platforms.
• Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness.
• Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.
• Develop and maintain scalable data processing solutions supporting analytics and machine learning workloads.

Required Qualifications

• Minimum 8 years of total IT experience.
• Minimum 4 years of recent experience working with Google Cloud Platform (GCP).
• Strong proficiency in Python and SQL, with experience in Scala and/or Java preferred.
• Strong expertise in Apache Spark, including Spark SQL, DataFrames, and Spark Streaming.
• Hands-on experience with streaming technologies such as Apache Kafka, Flink, or Pub/Sub.
• Experience designing and implementing batch and real-time data pipelines.
• Experience with GCP data services and cloud-based data engineering solutions.
• Knowledge of data warehousing technologies such as Snowflake and Redshift.
• Knowledge of NoSQL databases.
• Strong understanding of data quality, monitoring, and alerting practices.

Preferred Qualifications

• Experience with Azure data services.
• Experience with Apache Airflow.
• Experience with Databricks.
• Experience with Docker and Kubernetes.
• Experience optimizing data pipelines and distributed processing workloads for performance and cost efficiency.

Notes

Glider is needed for this role

Mandatory Areas

Must Have Skills – Data Engineer

• Skill 1 – big data, spark, pyspark, apache airflow, etl process
• Skill 2 – 4+ years in GCP

Good To have Skills –

• Skill 1 – Yrs of Exp –N/A
• Skill 2 – Yrs of Exp –N/A
• Skill 3 – Yrs of Exp –N/A
• Skill 4 – Yrs of Exp -N/A

Mandatory if Applicable

Domain Experience (If any ) – N/A

Must have Certifications –N/A

Onsite Requirement - Y/N- Y

Number of days onsite – 5 Days