Data platform engineer for autonomous vehicles

Grid Dynamics; Warszawa; On-site; PLN 200,000 - 360,000; Full time

Benefits offered by this job

Medical insurance

Flexible schedule

Corporate events

Well-equipped office

Job summary

Grid Dynamics приглашает инженера платформы данных для работы над основой данных и MLOps для систем безопасности автономной техники. Ваша задача — спроектировать и владеть многоуровневой AWS-платформой данных, которая питает обучение моделей и сбор телеметрии на масштабируемой инфраструктуре.

Вы реализуете medallion-архитектуру на AWS, адаптеры источников, реестр метаданных и схемы валидации. Участвуете в пилотном проекте и масштабировании для дальнейших автономных задач.

Qualifications

• 5+ лет опыта в построении производственных конвейеров данных и платформ.
• Глубокие знания S3, EventBridge, Step Functions, FSx for Lustre, CloudWatch и IAM; IaC и GitOps.
• Опыт Ray или Spark и Python для ETL/ELT на масштабе.
• Знание Parquet/HDF5; практический опыт версионирования наборов данных и lineage.
• Опыт MLflow или аналогичного реестра артефактов и трекинга экспериментов.
• Плюсом: автономные системы данные, ROS, LiDAR, камера и пр.

Responsibilities

• Спроектировать medallion-архитектуру данных на AWS S3 и внедрить правила продвижения между уровнями с IAM.
• Создать адаптерный слой для приведения источников к каноническому формату с сохранением временной метаданных и контекста датчиков.
• Развернуть централизованный реестр метаданных, версионирование наборов и развязку по уровням.
• Установить схемы валидации данных, концевые выключатели и дашборды для индикации ingestion/rejection и latency.
• Доказать полную цепочку инструментов от сбора данных до обучения моделей на пилотном кейсе (wheel-loader).

Skills

Data pipelines

AWS data platform

Python

SQL/ETL

IaC / GitOps

Event-driven arch

Data governance

MLflow / artifact tracking

Ray / Spark

Tools

S3

EventBridge

Step Functions

FSx for Lustre

CloudWatch

IAM

Job description

Описание:

We are seeking a Data Platform Engineer to join a co-innovation program building the data and MLOps foundation for collision mitigation safety systems on autonomous mining and construction vehicles, for a global heavy-equipment manufacturer and long-standing pioneer in autonomous machinery. In this role, you will work alongside ML teams to design and own a multi-tier AWS data platform that feeds safety and foundation model training, ingesting sensor telemetry, simulation outputs and annotated datasets at scale

Задачи:

Design and implement a medallion data architecture on AWS S3 with IAM-enforced tier promotion rules, replacing manual ingestion triggers with event-driven pipelines and data discovery logic that guarantees complete capture. Build an adapter layer that converts heterogeneous sources into a canonical format preserving temporal metadata, sensor calibration and spatial context, plus a gateway for large-scale real-world telemetry with automated PII scrubbing and sensor frequency alignment. Deploy a centralized metadata registry with immutable dataset versioning and branching per tier, enforce episode-level partitioning for strict train/validation splits, and establish provenance chains from raw data through processed datasets and training runs to model artifacts. Build automated schema validation gates and configurable circuit breakers that stop corrupted data from advancing between tiers, and a unified dashboard tracking ingestion and rejection rates, validation failures, augmentation throughput and latency. Integrate a data augmentation pipeline as a managed flow stage with full lineage from source data through augmentation parameters to output, and define with ML teams the contracts for schema completeness, temporal windowing, annotation quality thresholds and approval gates. Prove the end-to-end toolchain, from data ingestion to safety model training, on a first pilot workflow such as wheel-loader stockpile management, then scale it as the foundation for further autonomy use cases.

Требования:

• 5+ years of experience building production-grade data pipelines and data platforms.
• Deep expertise with S3, EventBridge, Step Functions, FSx for Lustre, CloudWatch and IAM, with IaC proficiency and GitOps practices.
• Hands-on experience with Ray or Spark and strong Python for building ETL/ELT pipelines at scale.
• Solid knowledge of columnar and binary formats (Parquet, HDF5) and event-driven ingestion architectures.
• Practical experience with LakeFS, DVC, Delta Lake or equivalent dataset versioning and lineage tools.
• Proven experience designing schema validation, circuit breakers and rejection-rate monitoring.
• Familiarity with MLflow or an equivalent artifact registry and experiment tracking system.
• Будет плюсом: Experience with autonomous systems data formats such as ROS bags, LiDAR, IMU and camera streams; Prior work with medallion or lakehouse architectures; Exposure to tools such as NVIDIA Cosmos, Stable Diffusion-based pipelines or similar; Experience in regulated or safety-critical data environments such as automotive, aerospace or industrial autonomy

Условия:

Opportunity to work on bleeding-edge projects Work with a highly motivated and dedicated team Competitive salary Flexible schedule Benefits package - medical insurance, sports Corporate social events Professional development opportunities Well-equipped office