Senior Data Engineer

Compunnel Inc.

Sunnyvale (CA)

On-site

USD 100,000 - 140,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Compunnel, Inc. is seeking a Data Engineer in Sunnyvale, California to build scalable, real-time data pipelines. The ideal candidate will work with technologies like Spark, Kafka, and GCP to facilitate data processing for analytics and machine learning.

Key responsibilities include designing ETL/ELT pipelines, optimizing data architectures, and ensuring data integrity. Applicants should have strong skills in Python, SQL, and Apache Spark, with at least 8 years of IT experience and specific expertise in GCP.

Qualifications

  • Strong proficiency in Python and SQL, with knowledge of Scala or Java.
  • 8+ years of total IT experience, with 4+ years in GCP.
  • Expertise in Apache Spark and streaming technologies.

Responsibilities

  • Design and develop ETL/ELT data pipelines using Spark and Kafka.
  • Build and optimize scalable data architectures.
  • Implement data quality checks and monitoring systems.

Skills

Python
Spark
Kafka
SQL
GCP

Tools

Airflow
Databricks
Docker
Kubernetes

Job description

Job Summary

Seeking a Data Engineer with Spark & Streaming skills to build real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML.

Key Responsibilities
  • Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies (Kafka, Flink).
  • Build and optimize scalable data architectures, including data lakes, data warehouses (BigQuery), and streaming platforms.
  • Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness.
  • Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.
Required Qualifications
  • Programming: Strong proficiency in Python, SQL, and potentially Scala/Java.
  • Big Data: Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
  • Streaming: Experience with messaging queues like Apache Kafka, or Pub/Sub.
  • Cloud: Familiarity with GCP, Azure data services.
  • Databases: Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.
  • Total IT Experience: Minimum 8 years.
  • GCP Experience: 4+ years of recent GCP experience.
Preferred Qualifications
  • Tools: Experience with Airflow, Databricks, Docker, Kubernetes.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Real-Time Data Engineer – Spark, Kafka & GCP
Real-Time Data Engineer – Spark, Kafka & GCP

Compunnel Inc. • Sunnyvale (CA)

On-site
USD 100,000 - 140,000
Senior Data Engineer
Senior Data Engineer

The Value Maximizer • South Carolina

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

EdgeQ Technologies Private Limited • Dallas (TX)

On-site
USD 100,000 - 160,000
Kafka Data Engineer
Kafka Data Engineer

Veriipro • Pennington (NJ)

On-site
USD 150,000 - 190,000
Senior Data Engineer: Real-Time Pipelines on AWS
Senior Data Engineer: Real-Time Pipelines on AWS

eOne Infotech • Fort Mill (SC)

On-site
USD 90,000 - 120,000
Senior Data Engineer
Senior Data Engineer

Peyton Resource Group • Houston (TX)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

Advanced Tech Placement • Atlanta (GA)

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Centillion Infotech LLC • Glendale (CA)

On-site
USD 120,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000