Lead Data Engineer (Databricks)

ACAISOFT POLAND Sp. z o.o.; Warszawa; On-site; PLN 180,000 - 280,000; Full time

Benefits offered by this job

Private medical care

Multisport card

Friendly, informal atmosphere, and in‑

Job summary

ACAISOFT POLAND Sp. z o.o. poszukuje Lead Data Architect do pracy w międzynarodowym środowisku, aby prowadzić architekturę danych i wspierać projekty BioPharma dla klientów z całego świata.

Rola obejmuje projektowanie modeli danych, budowę potoków oraz zapewnienie jakości i zaufania danych na skalę enterprise. Wymagane jest zaawansowane doświadczenie z Pythonem, SQL, Spark na Databricks, Delta Lake i Unity Catalog, a także praktyki IaC i CI/CD.

Qualifications

• Wymagane doświadczenie na stanowisku Lead/Principal Data Architect w środowiskach międzynarodowych.
• Głęboka znajomość modelowania danych, integracji, hurtowni danych i nowoczesnych platform danych.
• Języki Python i SQL oraz praktyczne doświadczenie z Spark na Databricks (Structured Streaming, Auto Loader, Delta Lake, Unity Catalog).
• Doświadczenie z danymi ustrukturyzowanymi i nieustrukturyzowanymi na pełnym cyklu: ingestion, transformacja, wzbogacanie, indeksowanie, retention, z praktykami IaC, CI/CD, testowanie potoków.
• Doświadczenie z integracją źródeł operacyjnych: Oracle, PostgreSQL, MongoDB; pattern CDC, schema drift, late data.

Responsibilities

• Budowanie, rozwijanie i utrzymanie możliwości wgrywania i przetwarzania danych strukturalnych, półstrukturalnych i nieustrukturalnych.
• Utrzymywanie metadata i praktyk jakości danych zapewniających cross-record querying i AI-ready access.
• Współpraca z architektami, inżynierami AI i ekspertami domenowymi celem używalności, wydajności i zaufania danych.
• Wsparcie jakości inżynierii poprzez przeglądy kodu i mentoring młodszych inżynierów.
• Prowadzenie decyzji architektonicznych i współpraca z interesariuszami biznesowymi i technologicznymi; kierowanie zespołami danych.

Skills

Python

SQL

Databricks

Spark

Data modelling

CI/CD

Infrastructure as code

Data integration

Tools

Delta Lake

Unity Catalog

Oracle

PostgreSQL

MongoDB

Job description

Nasze wymagania:

• Experience as a Lead / Principal Data Architect in multinational environments.
• Deep knowledge of data modelling, integration, warehousing, and modern data platforms.
• Strong hands-on Python and complex SQL, with production Spark experience on Databricks (Structured Streaming, Auto Loader, Delta Lake, Unity Catalog) to support analytics, AI/ML, and enterprise application use cases.
• Practice working with structured and unstructured data across the full lifecycle: ingestion, transformation, enrichment, indexing, and retention, delivered through disciplined engineering practice including infrastructure-as-code, CI/CD, and automated testing of pipelines.
• Proven experience ingesting from heterogeneous operational sources including relational (Oracle, PostgreSQL) and document stores (MongoDB), using Full load + CDC or equivalent replication patterns, with practical handling of schema drift, deletes, and late or out-of-order data.
• Skills in data modelling across a layered architecture, with clear contracts between raw, curated and serving layers, supporting cross-entity queries and contextual linking, and performant data access patterns.

Mile widziane:

• Experience building unstructured document pipelines: parsing and extraction from PDF, Office and scanned formats, chunking, embedding generation, and maintaining searchable indexes at scale.

O projekcie:

Hi there! If you're looking for a high-impact position in an ambitious software house, we've got a match for you! Currently, we are searching for a full-time Lead Data Engineer to cooperate with our US client - BioPharma organization. The mission is to unlock the potential of AI/ML to improve the lives of patients. Our client delivers powerful cloud software and services specifically designed to meet the evolving needs of the BioPharma sector. You will be an integral part of the hands-on delivery of reliable, scalable data pipelines and datasets that power AI discoverability, analytics, reporting, and workflow automation across scientific, clinical, and enterprise data. Due to the client's location, the team is working till 6:00 PM CEST.

Zakres obowiązków:

• Build, evolve, and operate data ingestion and processing capabilities for structured, semi-structured, and unstructured data, supporting the transition from early prototypes through early adoption and general release.
• Implement and maintain rich metadata and data quality practices that enable cross- record querying, traceability, and AI-ready data access across experiments, files, inventory, and workflows.
• Work with architects, AI engineers, workflow engineers, and domain experts to ensure data is usable, performant, and trustworthy for downstream GenAI and analytics use cases.
• Support engineering quality through code reviews and mentoring of junior/mid-level engineers.
• Lead architectural decisions and work with senior business and technology stakeholders.Guide and mentor data and engineering teams

Oferujemy:

• Private medical care
• Multisport card
• Friendly, informal atmosphere, and direct contact with everyone in the company
• Modern technologies