Senior Data Backend Engineer - Spark/Python/Scala (AWS/GCP)

Cloud Hybrid Technologies, LLC; Elk Grove (CA); Hybrid; USD 96,432,000 - 137,760,000; Full time

Job summary

A leading cloud technology firm is looking for a skilled Back-End Engineer with a strong data engineering background to design and develop scalable data pipelines. The ideal candidate will have experience in Spark, Python, Scala, and Java, and work with large datasets in a collaborative environment. Responsibilities include developing ETL processes, implementing cloud solutions, and ensuring data quality. This position offers a competitive hourly rate and is open for candidates in both the United States and India.

Qualifications

• 5+ years of experience in back-end development with a focus on data engineering.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient and optimized SQL queries for data extraction, transformation, and loading processes.
• Work with DataFrames to manipulate and analyze large datasets.
• Implement data storage and processing solutions using cloud technologies.
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Utilize S3 for data storage and retrieval.
• Work with data lake technologies like Iceberg.
• Ensure data quality, integrity, and security.
• Collaborate with data scientists and other engineers to understand data requirements.
• Participate in code reviews and contribute to improving development processes.
• Troubleshoot and resolve issues in data pipelines and back-end systems.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

AWS

GCP

MSK/Kafka

S3

Iceberg

data warehousing concepts

communication skills

collaboration skills

Education

Bachelor’s degree in computer science or a related field

Job description

A leading cloud technology firm is looking for a skilled Back-End Engineer with a strong data engineering background to design and develop scalable data pipelines. The ideal candidate will have experience in Spark, Python, Scala, and Java, and work with large datasets in a collaborative environment. Responsibilities include developing ETL processes, implementing cloud solutions, and ensuring data quality. This position offers a competitive hourly rate and is open for candidates in both the United States and India.