Databricks Data Engineer Consultant: Pipelines, Governance & ML

Chabre; Warszawa; On-site; PLN 198,000 - 331,000; Full time

Benefits offered by this job

Rate up to 240 PLN/h + VAT

Peripherals subsidy 500 PLN

Work tools provided

Course reimbursement

Job summary

Chabre IT Services is seeking a Data Engineer Consultant in Warsaw to design and implement data ingestion pipelines on the Databricks platform, connecting external APIs and business systems for scalable data flows.

You will develop Bronze and Silver layers, implement data quality controls, and support ML workflows, including feature stores and batch scoring. Excellent English and Polish (min B2) are required for successful collaboration on client projects.

Qualifications

• Recent, hands-on experience delivering production solutions with Databricks.
• Strong knowledge of PySpark, Delta Lake and Unity Catalog.
• Practical experience with the Databricks medallion architecture and production data pipelines.
• Experience with Spark Declarative Pipelines / Delta Live Tables and Databricks Workflows.
• Very good Python and advanced SQL skills.
• Experience integrating REST APIs into data engineering solutions, including authentication, pagination, rate limiting and incremental or checkpointed data retrieval.
• Strong understanding of data modelling, including SCD Type 2 and CDC patterns.
• Experience designing data quality controls, reconciliation processes and mechanisms for handling invalid or quarantined data.
• Knowledge of Git-based development and CI/CD for data platforms, preferably with Azure DevOps or GitHub Actions.
• Experience implementing monitoring and observability for production data pipelines.
• Ability to work within a fixed project scope, estimate work, meet milestones and take ownership of agreed deliverables.
• Strong communication skills and the ability to work independently in a consulting environment.
• Excellent English and Polish skills (min. B2 level).

Responsibilities

• Design and implement data ingestion pipelines connecting external APIs and business systems with the Databricks data platform.
• Develop reliable processes for retrieving and processing historical data, including mechanisms for rate limiting, checkpointing, retries and safe reprocessing.
• Build and maintain the Bronze layer of the data platform, covering data standardisation, typing, historical tracking and document storage.
• Develop processes for linking digital documents with corresponding business records while ensuring data consistency and idempotent processing.
• Use Databricks capabilities to classify documents and extract structured information from PDFs and images, including confidence scoring and workflows for manual review.
• Contribute to the development of the Silver layer using appropriate entity modelling, CDC and incremental processing patterns.
• Implement deterministic entity resolution and source-priority logic across multiple data sources.
• Apply data quality controls within the existing framework and ensure that quality results are properly stored, monitored and available for governance purposes.
• Support the development of analytical and machine-learning data products, including feature and metric stores and batch scoring workflows.
• Work with MLflow and related processes for model registration and promotion where required.
• Improve automation, monitoring and reliability across data pipelines and contribute to CI/CD processes.
• Collaborate with other technical stakeholders to deliver agreed project milestones within a defined consulting scope.

Skills

Databricks

PySpark/Delta

Unity Catalog

Delta Live Tables

Python/SQL

REST APIs

Data modelling (SCD2/CDC)

Data quality controls

CI/CD / Git

Monitoring/Observability

Project ownership

Communication (EN/PL)

English & Polish (B2)

Tools

Databricks

Git-based development

Azure DevOps/GitHub Actions

CI/CD pipelines

Job description

Chabre IT Services is seeking a Data Engineer Consultant in Warsaw to design and implement data ingestion pipelines on the Databricks platform, connecting external APIs and business systems for scalable data flows.

You will develop Bronze and Silver layers, implement data quality controls, and support ML workflows, including feature stores and batch scoring. Excellent English and Polish (min B2) are required for successful collaboration on client projects.