Senior Data Platform Engineer - Spark, Kafka, AWS

Robotics Technologies LLC; Sunnyvale (CA); On-site; USD 100,000 - 130,000; Full time

Job summary

A technology firm is seeking a skilled Back-End Engineer with a strong focus on data engineering to design and maintain data pipelines and back-end systems. The successful candidate will work with technologies like Spark, Python, and AWS. Responsibilities include writing SQL for ETL processes, implementing data solutions and collaborating with engineers. Candidates should have at least 5 years of experience in back-end development. This role is based in California, offering competitive compensation and opportunities for growth.

Qualifications

• 5+ years of experience in back-end development with a focus on data engineering.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient and optimized SQL queries for ETL processes.
• Work with DataFrames to manipulate and analyze large datasets.
• Implement data storage and processing solutions using cloud technologies.
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Utilize S3 for data storage and retrieval.
• Work with data lake technologies like Iceberg.
• Ensure data quality, integrity, and security.
• Collaborate with data scientists and engineers for solutions.
• Troubleshoot and resolve issues in data pipelines.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

AWS

GCP

MSK/Kafka

S3

Iceberg

Education

Bachelor’s degree in computer science or a related field

Job description

A technology firm is seeking a skilled Back-End Engineer with a strong focus on data engineering to design and maintain data pipelines and back-end systems. The successful candidate will work with technologies like Spark, Python, and AWS. Responsibilities include writing SQL for ETL processes, implementing data solutions and collaborating with engineers. Candidates should have at least 5 years of experience in back-end development. This role is based in California, offering competitive compensation and opportunities for growth.