Senior Data Platform Engineer

Machine Intelligence Technologies, LLC; Sunnyvale (CA); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Machine Intelligence Technologies LLC is seeking a skilled Back-End Engineer with a data engineering focus to design, develop, and maintain scalable data pipelines and back-end systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists and engineers across US and India.

The role requires expertise in Spark, Python, Scala, Java, SQL, and data frames, plus experience with cloud platforms like AWS or GCP and streaming with Kafka.

Qualifications

• Bachelor’s degree in computer science or a related field or equivalent experience.
• 5+ years of backend development focusing on data engineering.
• Strong communication and collaboration skills.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient SQL queries for ETL processes.
• Work with DataFrames to manipulate and analyze large datasets.
• Implement data storage and processing using cloud technologies (AWS or GCP).
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Utilize S3 for data storage and retrieval.
• Work with data lake technologies like Iceberg.
• Ensure data quality, integrity, and security.
• Collaborate with data scientists and engineers to understand data requirements.
• Participate in code reviews and improve development processes.
• Troubleshoot and resolve issues in data pipelines and back-end systems.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

AWS

GCP

Kafka

S3

Iceberg

Data Warehousing

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

AWS

GCP

Kafka

S3

Iceberg

Job description

Machine Intelligence Technologies LLC is seeking a skilled Back-End Engineer with a data engineering focus to design, develop, and maintain scalable data pipelines and back-end systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists and engineers across US and India.

The role requires expertise in Spark, Python, Scala, Java, SQL, and data frames, plus experience with cloud platforms like AWS or GCP and streaming with Kafka.