Senior Data Engineer

Compunnel, Inc.; Sunnyvale (CA); On-site; USD 100,000 - 140,000; Full time

Job summary

Compunnel, Inc. is seeking a Data Engineer in Sunnyvale, California to build scalable, real-time data pipelines. The ideal candidate will work with technologies like Spark, Kafka, and GCP to facilitate data processing for analytics and machine learning.

Key responsibilities include designing ETL/ELT pipelines, optimizing data architectures, and ensuring data integrity. Applicants should have strong skills in Python, SQL, and Apache Spark, with at least 8 years of IT experience and specific expertise in GCP.

Qualifications

• Strong proficiency in Python and SQL, with knowledge of Scala or Java.
• 8+ years of total IT experience, with 4+ years in GCP.
• Expertise in Apache Spark and streaming technologies.

Responsibilities

• Design and develop ETL/ELT data pipelines using Spark and Kafka.
• Build and optimize scalable data architectures.
• Implement data quality checks and monitoring systems.

Skills

Python

Spark

Kafka

SQL

GCP

Tools

Airflow

Databricks

Docker

Kubernetes

Job description

Job Summary

Seeking a Data Engineer with Spark & Streaming skills to build real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies (Kafka, Flink).
• Build and optimize scalable data architectures, including data lakes, data warehouses (BigQuery), and streaming platforms.
• Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness.
• Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.

Required Qualifications

• Programming: Strong proficiency in Python, SQL, and potentially Scala/Java.
• Big Data: Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
• Streaming: Experience with messaging queues like Apache Kafka, or Pub/Sub.
• Cloud: Familiarity with GCP, Azure data services.
• Databases: Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.
• Total IT Experience: Minimum 8 years.
• GCP Experience: 4+ years of recent GCP experience.

Preferred Qualifications

• Tools: Experience with Airflow, Databricks, Docker, Kubernetes.