data engineer in pharma/biotech

NDA; United States; Remote; USD 18,000 - 27,000; Full time

Job summary

NDA ищет старшего инженера по данным для проектирования и поддержки конвейеров на Azure Databricks: Landing, Raw, Enriched, Curated. Роль предполагает создание архитектуры, надзор за их работоспособностью и соответствием требованиям.

Кандидат должен иметь практический опыт Databricks Jobs, Delta Lake и dbt, обеспечивать качество данных, управление версиями в рамках Dev/Test/Prod, а также участие в CI/CD и MLOps на Databricks.

Qualifications

• Опыт работы с Azure Databricks, Spark и Delta Lake.
• Опыт построения многослойных конвейеров (Landing, Raw, Enriched, Curated).
• Опыт организации пайплайнов через Databricks Jobs/Workflows с зависимостями и оповещениями.
• Опыт контроля качества данных на входе, обработки дрейфа схем.
• Опыт dbt для наблюдаемости пайплайнов и управления данными.

Responsibilities

• Разработка и сопровождение многоуровневых конвейеров на Azure Databricks.
• Оркестрация задач через Databricks Jobs/Workflows, обработка ошибок и оповещения.
• Внедрение пакетной и API-интеграций.
• Контроль качества данных, соблюдение SLA.
• Работа в средах Dev/Test/Prod, включая доступ и аудит.
• Вклад в CI/CD и MLOps на Databricks.

Skills

Azure Databricks

Spark

Delta Lake

dbt

Data pipelines

Databricks Jobs

Data quality checks

ETL/ELT

CI/CD for data workflows

Job description

Задачи

• Build and maintain multi-layer pipelines (Landing Raw Enriched Curated) on Azure Databricks using Spark and Delta Lake
• Orchestrate pipelines via Databricks Jobs/Workflows, including dependencies, retries, and alerting
• Implement batch and API-based data ingestion
• Apply schema enforcement, handle schema drift, and run data quality checks at ingestion
• Use dbt for pipeline observability and data governance
• Monitor job failures, data freshness, and pipeline health against SLA targets
• Set up alerts, logging, and retry strategies for production pipelines
• Work within controlled Dev/Test/Prod environments, including access control, audit trails, pipeline documentation, and traceability to requirements
• Contribute to CI/CD and MLOps/AI pipelines on Databricks

Требования

• Strong hands-on experience with Azure Databricks, Spark, and Delta Lake
• Experience with multi-layer pipelines (Landing Raw Enriched Curated)
• Experience with schema enforcement and schema-drift handling
• Experience with data quality checks at ingestion
• Experience orchestrating pipelines via Databricks Jobs/Workflows, including dependencies, retries, and alerts
• Experience with batch and API-based data ingestion
• Strong hands-on experience with dbt
• Experience monitoring job failures, data freshness, and pipeline health
• Experience with SLA-based data delivery
• Experience with alerts, logging, and retry strategies
• Knowledge of access control models, audit trails, lineage, and traceability
• Experience with data contracts, pipeline documentation, and controlled environments (Dev/Test/Prod)
• EU residency required; candidates outside the European Union are not considered
• English proficiency: B2+ Будет плюсом: Git-based development, CI/CD pipelines for data workflows, MLOps/AI pipelines on Databricks, exposure to GxP, clinical trials, or pharmacovigilance, strong SQL engineering and modular reusable dbt models, dimensional modeling (star schema, OBT, curated marts), Tabular Editor, Power BI, pharma/biotech industry experience

Условия

• Long-term project with a Danish client
• Senior, independent role with minimal oversight
• Focus on a modern data stack (Databricks, dbt, Delta Lake)