Sr Data Engineering (Foundation Models) - Only W2

Avacend Inc; San Jose (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

A technology company in San Jose is looking for a skilled data engineer to design and scale data pipelines essential for model training and production systems. The ideal candidate should have over 5 years of software engineering experience, strong proficiency in Python, and hands-on expertise with Apache Spark. Responsibilities include building distributed data pipelines, optimizing performance, and collaborating with ML engineers. A strong emphasis on debugging and data quality is essential for success in this role.

Qualifications

• 5+ years of software engineering experience.
• Strong proficiency in Python.
• Experience working with time series, logs, or high-volume event data.
• Excellent debugging and performance optimization skills.

Responsibilities

• Build and scale distributed data pipelines for large-scale time series and log data.
• Design reliable, high-performance Spark/Python workflows for model training datasets.
• Analyze and resolve performance bottlenecks.
• Improve data quality, validation, and reproducibility for ML workloads.
• Collaborate with ML researchers to accelerate foundation model development.
• Measure and optimize application performance in production systems.

Skills

Python

Apache Spark

Debugging skills

Performance optimization

Tools

Spark

Kafka

Kubernetes

Job description

We are building next-generation foundation models for machine-generated data — including time series, logs, and large-scale event streams. We’re looking for a strong data engineer to design and scale the data pipelines that power model training and production systems.

This role sits at the intersection of distributed systems, data infrastructure, and machine learning.

What You’ll Do

• Build and scale distributed data pipelines for large-scale time series and log data
• Design reliable, high-performance Spark/Python workflows for model training datasets
• Analyze and resolve performance bottlenecks (latency, memory, skew, throughput)
• Improve data quality, validation, and reproducibility for ML workloads
• Partner with ML engineers and researchers to accelerate foundation model development
• Measure and optimize application and transaction performance in production systems
• 5+ years of software engineering experience
• Strong proficiency in Python
• Hands‑on experience with Apache Spark (PySpark or Scala)
• Experience working with time series, logs, or high‑volume event data
• Strong debugging and performance optimization skills

Nice to Have

• Experience supporting ML or large model training workflows
• Familiarity with sequence modeling or time series data systems
• Experience with streaming systems (Kafka, Spark Streaming)
• Experience with cloud‑native or Kubernetes‑based platforms

What We’re Looking For

You are pragmatic, data‑driven, and comfortable operating in ambiguous, research‑heavy environments. You can debug distributed systems, reason about data correctness at scale, and build infrastructure that researchers trust.