Senior Data Engineer – Vision Data Platform

CADS GmbH; Österreich; Vor Ort; EUR 70.000 - 110.000; Vollzeit

Zusammenfassung

CADS GmbH is seeking a skilled data platform engineer to design and operate production-grade pipelines for clinical images and metadata. You will architect scalable storage and lineage, validate data, and partner with ML teams to deliver versioned datasets.

You will implement end-to-end workflows from ingestion to release, ensure compliance, and maintain observability and auditable data handling across the platform. Austria-based role with potential hybrid elements.

Qualifikationen

• Experience designing and operating production-grade data pipelines for reliable ingestion, transformation, and delivery of structured and unstructured data.
• Strong production experience with Python and SQL.
• Hands-on experience with AWS services for data platforms (S3, IAM, CloudWatch, SQS/SNS, ECS/EKS, Lambda, RDS/PostgreSQL).
• Experience designing and operating production-grade data pipelines for both structured and unstructured data.
• Experience with object storage and large binary assets such as images, documents, media, or similar unstructured data.
• Experience with workflow orchestration tools such as Prefect, Dagster, or Airflow.
• Experience with PostgreSQL or comparable systems for operational metadata, integrating REST APIs and building automation.
• Experience implementing data validation, data quality controls, observability, and monitoring in production systems.

Aufgaben

• Design and build scalable data pipelines that can reliably handle high volumes of clinical images and metadata from hospital and application workflows.
• Create strong data validation and quality processes to ensure images and metadata are complete, accurate, compliant, and traceable, while handling duplicates and failures.
• Develop the underlying data architecture for large image assets and operational metadata, including storage design, canonical data models, lineage, and curated datasets for ML use cases.
• Build and manage end-to-end workflows from ingestion through validation, pre-annotation, annotation-platform integration, curation, to final dataset release with reliability and traceability.
• Work closely with annotation and ML teams to deliver high-quality, versioned datasets with compliant data handling and monitor/auditability across the platform.

Kenntnisse

ETL/ELT design

Python

SQL

AWS

Data pipelines

Object storage

Workflow orchestration

PostgreSQL

Data validation & observability

Tools

AWS

Airflow

Prefect

Dagster

PostgreSQL

Jobbeschreibung

Qualifications

• Experience designing and managing ETL/ELT processes for reliable ingestion, transformation, and delivery of structured and unstructured data.
• Strong production experience with Python and SQL
• Strong hands‑on experience with AWS, especially services relevant to data platforms such as S3, IAM, CloudWatch, SQS/SNS, ECS/EKS, Lambda, and RDS/PostgreSQL
• Experience designing and operating production‑grade data pipelines for both structured and unstructured data
• Experience with object storage and large binary assets such as images, documents, media, or similar unstructured data
• Experience with workflow orchestration tools such as Prefect, Dagster, or Airflow
• Experience with Postgre SQL or comparable systems for operational metadata, integrating REST APIs and building automation around external platforms or internal services
• Experience implementing data validation, data quality controls, observability, and monitoring in production systems

Your Tasks

• Design and build scalable data pipelines that can reliably handle high volumes of clinical images and metadata coming from hospital and application workflows.
• Create strong data validation and quality processes to ensure images and metadata are complete, accurate, compliant, and easy to trace, while also handling duplicates, failures, and reprocessing when needed.
• Develop the underlying data architecture for large image assets and operational metadata, including storage design, canonical data models, lineage, and curated datasets that can support machine learning use cases.
• Build and manageend-to-end workflows that take data from ingestion through validation, pre‑annotation, annotation‑platform integration, curation, and final dataset release in a reliable and measurable way.
• Work closely with annotation and ML teams to deliver high-quality, versioned, and traceable datasets, while maintaining strong monitoring, auditability, and compliance‑aware data handling across the platform.