Principal Data Engineer - Backend & Pipelines

Cloud Hybrid Technologies, LLC; Sunnyvale (CA); Hybrid; USD 110,000 - 150,000; Full time

Job summary

A leading technology company is seeking a skilled Back-End Engineer to design and maintain data pipelines in a dynamic environment. The role involves working with Spark, Python, and various cloud technologies. The ideal candidate will have over 5 years of experience in back-end development and a comprehensive understanding of data engineering principles. Collaboration with data teams and effective communication are essential to delivering impactful solutions. This role is based in California, with potential opportunities in India.

Qualifications

• 5+ years of experience in back-end development focusing on data engineering.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient SQL queries for ETL processes.
• Work with DataFrames to manipulate large datasets.
• Implement cloud data storage and processing solutions.
• Build real-time data streaming pipelines using MSK/Kafka.
• Utilize S3 for data storage and retrieval.
• Ensure data quality and integrity.
• Collaborate with teams to deliver solutions.
• Participate in code reviews to improve processes.
• Troubleshoot issues in data pipelines.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

AWS

GCP

MSK/Kafka

S3

Iceberg

Education

Bachelor’s degree in computer science or related field

Job description

A leading technology company is seeking a skilled Back-End Engineer to design and maintain data pipelines in a dynamic environment. The role involves working with Spark, Python, and various cloud technologies. The ideal candidate will have over 5 years of experience in back-end development and a comprehensive understanding of data engineering principles. Collaboration with data teams and effective communication are essential to delivering impactful solutions. This role is based in California, with potential opportunities in India.