Principal Data Engineer — Cloud, Spark & Streaming

Quantum Technologies. LLC; Sunnyvale, Northern (CA, KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Quantum Technologies LLC in Sunnyvale is looking for a Back-End Engineer with a strong data engineering background to design, develop, and maintain scalable data pipelines and backend systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists and engineers to deliver impactful data-driven solutions.

Responsibilities include building data pipelines with Spark, Python, Scala, and Java; writing optimized SQL; working with DataFrames; using AWS or

Qualifications

• Bachelor's degree in computer science or a related field is required or equivalent experience.
• 5+ years of back-end development with a focus on data engineering.
• Strong proficiency in Spark, Python, Scala, and Java.
• Expertise in SQL and working with relational databases.
• Experience with DataFrames for data manipulation and analysis.
• Experience with cloud technologies (AWS or GCP).
• Experience with MSK/Kafka for streaming data.
• Experience with S3 or similar object storage.
• Experience with data lake technologies like Iceberg.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient and optimized SQL queries for ETL processes.
• Work with DataFrames to manipulate and analyze large datasets.
• Implement data storage and processing solutions using cloud technologies (AWS or GCP).
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Utilize S3 for data storage and retrieval.
• Work with data lake technologies like Iceberg.
• Ensure data quality, integrity, and security.
• Collaborate with data scientists and other engineers to understand data requirements.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

Cloud AWS/GCP

Kafka/MSK

S3

Iceberg

Data warehousing

Education

Bachelor's degree in Computer Science or related field

Tools

MSK/Kafka

S3

Iceberg

AWS

GCP

Job description

Quantum Technologies LLC in Sunnyvale is looking for a Back-End Engineer with a strong data engineering background to design, develop, and maintain scalable data pipelines and backend systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists and engineers to deliver impactful data-driven solutions.

Responsibilities include building data pipelines with Spark, Python, Scala, and Java; writing optimized SQL; working with DataFrames; using AWS or