Real-Time Data Engineer – Spark, Kafka & GCP

Compunnel, Inc.; Sunnyvale (CA); On-site; USD 100,000 - 140,000; Full time

Job summary

Compunnel, Inc. is seeking a Data Engineer in Sunnyvale, California to build scalable, real-time data pipelines. The ideal candidate will work with technologies like Spark, Kafka, and GCP to facilitate data processing for analytics and machine learning.

Key responsibilities include designing ETL/ELT pipelines, optimizing data architectures, and ensuring data integrity. Applicants should have strong skills in Python, SQL, and Apache Spark, with at least 8 years of IT experience and specific expertise in GCP.

Qualifications

• Strong proficiency in Python and SQL, with knowledge of Scala or Java.
• 8+ years of total IT experience, with 4+ years in GCP.
• Expertise in Apache Spark and streaming technologies.

Responsibilities

• Design and develop ETL/ELT data pipelines using Spark and Kafka.
• Build and optimize scalable data architectures.
• Implement data quality checks and monitoring systems.

Skills

Python

Spark

Kafka

SQL

GCP

Tools

Airflow

Databricks

Docker

Kubernetes

Job description

Compunnel, Inc. is seeking a Data Engineer in Sunnyvale, California to build scalable, real-time data pipelines. The ideal candidate will work with technologies like Spark, Kafka, and GCP to facilitate data processing for analytics and machine learning.

Key responsibilities include designing ETL/ELT pipelines, optimizing data architectures, and ensuring data integrity. Applicants should have strong skills in Python, SQL, and Apache Spark, with at least 8 years of IT experience and specific expertise in GCP.