Data engineer in observability platforms

Rapinno Tech Solutions GmbH; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

Rapinno Tech Solutions GmbH ищет старшего Data Engineer для проекта DEPOT observability. Вы будете проектировать и внедрять масштабируемые конвейеры данных, модели и трансформации, обеспечивая мониторинг качества и обнаружение аномалий.

Роль предполагает руководство архитектурой данных, наставничество, взаимодействие с командами Platform/AI/DevOps и участие в документации по проекту для обеспечения доступности данных во всей экосистеме.

Qualifications

• Опыт проектирования производственных конвейеров данных и ELT/ETL в корпоративной среде.
• Глубокие знания SQL и Python для трансформаций и автоматизации.
• Экспертиза в Apache Spark (PySpark/Scala) и Apache Airflow.
• Опыт работы с dbt, Kafka, облачными платформами и хранилищами данных.
• Навыки контроля качества данных и мониторинга через инструменты observability.

Responsibilities

• Проектирование и внедрение масштабируемых конвейеров данных для DEPOT.
• Разработка трансформаций, моделей данных и методов инкрементной загрузки.
• Оркестрация потоков с Airflow, настройка мониторинга и алертинга.
• Работа с Spark, Kafka, Snowflake/Databricks и облачными платформами.
• Сотрудничество с командами Platform, AI и DevOps; участие в ревью кода.

Skills

SQL

Python

PySpark

dbt

Airflow

Kafka

Spark

Cloud platforms

Git

CI/CD

Docker/Kubernetes

Data observability tools

Tools

Snowflake

Databricks

BigQuery

Azure Synapse

Kafka

Airflow

dbt

Great Expectations

Monte Carlo

Soda

Job description

Описание:

The Data Engineer (Senior) is a core contributor to the Data Observability RFQ — DEPOT platform. In this senior‑level role, you will architect and implement scalable data pipelines, data models, and transformation logic that power the DEPOT observability platform, enabling real‑time data quality monitoring and anomaly detection across enterprise data ecosystems. You will drive data architecture decisions, establish engineering best practices, and serve as a technical mentor to junior engineers within the team.

Задачи:

• Design, build, and optimize scalable data ingestion pipelines and ELT/ETL workflows that feed the DEPOT observability platform, ensuring high throughput, reliability, and data quality at scale.
• Develop and maintain data transformation logic using dbt and SQL, implementing dimensional models, data vault patterns, and incremental load strategies across cloud data warehouses and lakehouses.
• Architect and manage real‑time and batch data streaming workflows using Apache Kafka and Apache Spark, integrating diverse enterprise data sources into the DEPOT pipeline ecosystem.
• Own pipeline orchestration using Apache Airflow — designing DAG structures, managing dependencies, implementing alerting, and continuously improving pipeline observability and resiliency.
• Collaborate with platform engineers, AI engineers, and DevOps teams to integrate data engineering components into the broader DEPOT observability platform and ensure seamless data availability.
• Lead data architecture discussions, contribute to technical design documents, and participate in code reviews to uphold engineering standards and data modeling best practices across the team.
• Instrument data quality checks, freshness monitors, and lineage tracking across pipelines, partnering with observability tooling (e.g., Great Expectations, Monte Carlo) to proactively detect and resolve data issues.
• Mentor junior data engineers through code reviews, pair programming, and knowledge‑sharing sessions; contribute to internal documentation, reusable templates, and DataOps tooling.

Требования:

• Proven experience designing and implementing production‑grade data pipelines, ELT/ETL workflows, and data models in enterprise environments
• Deep proficiency in SQL and Python for data transformation, pipeline scripting, and automation
• Strong expertise with Apache Spark (PySpark/Scala)
• Strong expertise with Apache Airflow (DAG design, operators, sensors)
• Strong expertise with dbt (models, tests, macros, incremental strategies)
• Strong expertise with Apache Kafka (producers/consumers, topic management, streaming patterns)
• Hands‑on experience with cloud‑native data platforms such as Snowflake, Databricks, BigQuery, or Azure Synapse
• Familiarity with cloud infrastructure services (AWS, Azure, or GCP) for storage, compute, and data integration
• Experience with version control (Git)
• Experience with CI/CD practices for data workflows
• Experience with containerized environments (Docker/Kubernetes)
• Experience with data observability or data quality tooling (Great Expectations, Monte Carlo, Soda, or similar)
• Будет плюсом: Experience working within a data observability platform context — instrumenting metadata, lineage, freshness, and quality metrics across pipelines; Familiarity with DataOps frameworks, automated data testing, and schema evolution management in production environments; Exposure to advanced Kafka ecosystem tooling (Kafka Streams, ksqlDB, Confluent Platform), event‑driven architecture patterns, and real‑time analytics use cases; Knowledge of data lakehouse architectures (Delta Lake, Apache Iceberg, or Apache Hudi); Experience with infrastructure‑as‑code (Terraform, Pulumi) for data platform provisioning

Условия:

Contract Fully remote work