Remote Spark Data Engineer for AI & Big Data Platform

Addepto; Town of Poland (NY); Hybrid; USD 110,000 - 170,000; Full time

Benefits offered by this job

Flexible remote or office work

Professional training and conferences

Paid time off

Formal onboarding buddy

Medical and wellness benefits

Job summary

Addepto is seeking a data platform engineer to build and maintain a high‑performance data processing platform for automotive data. You will design scalable pipelines for streaming and batch workloads using Spark, Cloudera, Airflow, and Iceberg, and collaborate with cross‑functional teams to meet data requirements.

You will also optimize data workflows on AWS, write production-grade Python or Java/Scala code, and help ensure reliability, availability, and accurate analytics for enterprise clients.

Qualifications

• 4+ years of commercial experience with Big Data systems and data management.
• Strong programming in Python or Java/Scala with clean OO design.
• Hands-on with Spark, Cloudera, Airflow, Iceberg and related data platforms.
• Cloud experience in AWS and data processing in production environments.
• Consulting experience and excellent client-facing communication.

Responsibilities

• Develop and maintain a high-performance data processing platform for automotive data.
• Design and implement data pipelines for streaming and batch processing.
• Optimize workflows for ingestion, processing, and storage using Spark, Cloudera, Airflow.
• Work with data lake technologies (e.g., Iceberg) for structured/unstructured data.
• Collaborate with cross-functional teams to integrate data sources.
• Monitor and troubleshoot to ensure high availability and accuracy.
• Leverage AWS for infrastructure and scaling of workloads.
• Write high-quality Python or Java/Scala code for data tasks and automation.

Skills

Python

Java/Scala

Big Data

Cloud AWS

Spark

Cloudera

Airflow

Iceberg

Kafka

Education

Bachelor's degree in technical or mathematical studies

Tools

Databricks

dbt

Job description

Addepto is seeking a data platform engineer to build and maintain a high‑performance data processing platform for automotive data. You will design scalable pipelines for streaming and batch workloads using Spark, Cloudera, Airflow, and Iceberg, and collaborate with cross‑functional teams to meet data requirements.

You will also optimize data workflows on AWS, write production-grade Python or Java/Scala code, and help ensure reliability, availability, and accurate analytics for enterprise clients.