Senior Backend Engineer: Spark, Python, Scala on AWS/GCP

MACHINE LEARNING TECHNOLOGIES LLC; Elk Grove (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC is seeking a Senior Backend Engineer to design and implement scalable data pipelines and back-end systems. You will work with Spark, Python, Scala, and Java, and leverage cloud platforms to process large datasets.

You will build real-time streaming with MSK/Kafka, utilize S3 storage, and ensure data quality and security while collaborating with data scientists and other engineers. Experience in data warehousing and CI/CD is a plus.

Qualifications

• Bachelor’s degree in computer science or a related field or equivalent experience.
• 5+ years of backend data engineering experience.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient SQL for ETL processes and data extraction.
• Work with DataFrames to manipulate large datasets and ensure data quality.
• Implement cloud-based storage and processing using AWS or GCP.
• Build real-time data streaming pipelines with MSK/Kafka and S3 data lake technologies.
• Collaborate with data scientists and engineers; participate in code reviews and CI/CD.
• Troubleshoot data pipelines and back-end systems to ensure reliability.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

AWS

GCP

Kafka MSK

S3

Iceberg

Education

Bachelor’s degree in computer science or related field

Tools

Apache Kafka

CI/CD

Docker

Kubernetes

AWS

GCP

Job description

MACHINE LEARNING TECHNOLOGIES LLC is seeking a Senior Backend Engineer to design and implement scalable data pipelines and back-end systems. You will work with Spark, Python, Scala, and Java, and leverage cloud platforms to process large datasets.

You will build real-time streaming with MSK/Kafka, utilize S3 storage, and ensure data quality and security while collaborating with data scientists and other engineers. Experience in data warehousing and CI/CD is a plus.