Senior Data Backend Engineer - Spark, AWS & Kafka

Artifintech; Sunnyvale (CA); Remote; USD 140,000 - 190,000; Full time

Job summary

ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC is seeking a Back-End Engineer with a strong data engineering background to design, develop, and maintain scalable data pipelines and back-end systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists to deliver data-driven solutions.

You will design data pipelines using Spark, Python, Scala, and Java, write optimized SQL for ETL, and work with DataFrames.

Qualifications

• Bachelor’s degree or equivalent experience.
• 5+ years of experience in back-end development with a focus on data engineering.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient and optimized SQL queries for ETL processes.
• Work with DataFrames to manipulate and analyze large datasets.
• Implement data storage and processing solutions using cloud technologies (AWS or GCP).
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Utilize S3 for data storage and retrieval.
• Work with Iceberg and data lake technologies.
• Ensure data quality, integrity, and security.
• Collaborate with data scientists and other engineers to understand data requirements and deliver solutions.
• Participate in code reviews and improve development processes.
• Troubleshoot and resolve issues in data pipelines and back-end systems.

Skills

Spark

Python

Scala

Java

SQL

DataFrames

Cloud AWS/GCP

Kafka

S3

Iceberg

Data warehousing

Communication

Education

Bachelor’s degree in Computer Science or related field

Tools

MSK

Job description

ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC is seeking a Back-End Engineer with a strong data engineering background to design, develop, and maintain scalable data pipelines and back-end systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists to deliver data-driven solutions.

You will design data pipelines using Spark, Python, Scala, and Java, write optimized SQL for ETL, and work with DataFrames.