Senior Data Engineering Architect - Spark & Cloud Pipelines

Quantum Technologies. LLC; Sunnyvale (CA); Hybrid; USD 120,000 - 150,000; Full time

Job summary

A technology company based in Sunnyvale, California, seeks a skilled Back-End Engineer to design and maintain scalable data pipelines. The ideal candidate will have over 5 years of experience in back-end development and proficiency in Spark, Python, Scala, and Java. The role also involves collaborating with data scientists to deliver data-driven solutions while ensuring data integrity and quality across systems. This position offers a competitive salary and is inclusive of diverse candidates.

Qualifications

• 5+ years of experience in back-end development with a focus on data engineering.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient and optimized SQL queries for data ETL processes.
• Work with DataFrames to manipulate and analyze large datasets.
• Implement cloud storage and processing solutions.
• Build and maintain real-time data streaming pipelines.
• Ensure data quality, integrity, and security.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

AWS

GCP

MSK/Kafka

S3

Iceberg

Data warehousing

Communication

Collaboration

Education

Bachelor’s degree in computer science or related field

Job description

A technology company based in Sunnyvale, California, seeks a skilled Back-End Engineer to design and maintain scalable data pipelines. The ideal candidate will have over 5 years of experience in back-end development and proficiency in Spark, Python, Scala, and Java. The role also involves collaborating with data scientists to deliver data-driven solutions while ensuring data integrity and quality across systems. This position offers a competitive salary and is inclusive of diverse candidates.