Data Engineer, Real-Time ETL for Connected Vehicles

Viaduct; United States; Remote; USD 110,000 - 180,000; Full time

Benefits offered by this job

Kubernetes experience

Experience with ML teams

Open source contributions

Automotive industry exposure

Small, agile teams

Job summary

Viaduct is seeking an early data engineer to build and maintain an end-to-end ETL platform for automotive analytics and ML workloads. You will ensure data quality, reliability, and scalability from ingestion to client integrations, collaborating with ML and data science teams.

The role emphasizes strong Spark, Python/Scala/Java/C++, and distributed data systems experience. You will mentor teammates and help shape data pipelines in a fast-paced, open-source–friendly environment.

Qualifications

• 3+ years as a data engineer.
• Experience as a tech lead or mentor.
• Proficiency in Python/Scala/Java/C++ and SQL.
• 3+ years of experience with Spark or equivalent technologies.
• 2+ years of experience with a workflow scheduler (Airflow, Prefect, Argo, etc).
• 2+ years of experience with distributed file-systems (HDFS, S3, etc).
• Familiar with the open-source data ecosystem (Apache, CNCF, etc).
• Experience with incremental or real-time processing (Delta Lake, Apache Hudi, Kafka Stream, Spark Streaming).

Responsibilities

• Creating and supporting batch, incremental, and real-time ETL pipelines.
• Standardizing ingestion, validation, and cleaning processes across clients.
• Automating data validation to increase data quality.
• Managing and evolving schemas in all parts of our pipeline.
• Monitoring, tuning, and optimizing Spark jobs.
• Becoming a domain‑expert in connected vehicle data.

Skills

Data engineering

Python/Scala/Java/C++

SQL

Spark

Airflow/Prefect/Argo

Distributed file systems

Tools

Delta Lake

Apache Hudi

Kafka

Spark Streaming

HDFS/S3

Job description

Viaduct is seeking an early data engineer to build and maintain an end-to-end ETL platform for automotive analytics and ML workloads. You will ensure data quality, reliability, and scalability from ingestion to client integrations, collaborating with ML and data science teams.

The role emphasizes strong Spark, Python/Scala/Java/C++, and distributed data systems experience. You will mentor teammates and help shape data pipelines in a fast-paced, open-source–friendly environment.