Senior Data Engineer — Scalable Pipelines & AI Training

Versa Networks; Santa Clara (CA); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

ESOP

Relocation benefits

Flexi working

Bonuses & incentives

Job summary

You will implement observability, CI/CD practices, and data quality checks while shaping data models for AI/ML training and analytics initiatives. This role offers hybrid work options and opportunities to impact large-scale datasets.

Qualifications

• 3-5 years of experience designing and operating production data pipelines at scale.
• Proficient in Python for data engineering tasks.
• Experience with Airflow DAGs and containerized workflows.
• Experience with cloud platforms (GCP preferred) and BigQuery/Cloud Storage.

Responsibilities

• Design, build, and maintain production-grade data pipelines for terabytes of data.
• Collaborate with data scientists and software engineers to ensure scalable, reliable data infrastructure.
• Architect and deploy batch and streaming pipelines using Airflow and containerized workflows.
• Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm.
• Build high-throughput data transformations using Dask or Apache Spark.
• Maintain training data clusters across hybrid environments (on-prem and cloud).
• Implement observability (logging, metrics, alerting) to monitor pipeline health and SLA adherence.
• Troubleshoot data-processing failures in production and resolve issues quickly.
• Define data contracts, schemas, and quality checks with cross-functional teams.
• Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation.
• Design and maintain data models and schemas for AI/ML training use cases.
• Load data into cloud storage and lakes, ensuring performance and accessibility.

Skills

Python

Data pipelines

Collaboration

Tools

Airflow

Docker

Kubernetes

Helm

Spark

Dask

Ray

Terraform

Go

Rust

GCP

BigQuery

Cloud Storage

GKE

Job description

You will implement observability, CI/CD practices, and data quality checks while shaping data models for AI/ML training and analytics initiatives. This role offers hybrid work options and opportunities to impact large-scale datasets.