Senior Databricks Engineer

Capgemini Polska; Warszawa; Hybrid; PLN 180,000 - 280,000; Full time

Benefits offered by this job

Bonus roczny

Prywatna opieka medyczna Medicover

Ubezpieczenie na życie

Dostęp do platformy NAIS i szkoleń

Job summary

Capgemini Polska poszukuje inżyniera danych z doświadczeniem w Databricks; obejmuje pracę nad tworzeniem lakehouse i end-to-end pipelines, z naciskiem na architekturę i rozwiązywanie problemów klienta.

Wymagana jest biegła znajomość Spark, SQL i Python, a także doświadczenie z chmurami (Azure preferowane). Praca hybrydowa z udziałem zespołu w Warszawie i klientów międzynarodowych.

Qualifications

• Co najmniej 3 lata doświadczenia w data engineering (poziom Engineer) lub 5+ lat (Senior Engineer).
• Doświadczenie z Databricks w produkcji (minimum rok).
• Silna znajomość Spark (PySpark/Scala) i SQL; Python.

Responsibilities

• Budowa platform lakehouse na Databricks end-to-end: workspace, ingestion, pipelines, model danych, produkty danych dla analiz i ML oraz operacje utrzymania.
• Praca na całym stacku: od plattformy po pipelines; prowadzenie technicznego designu w zespole.
• W roli senior: projektowanie architektury, przegląd prac innych inżynierów, kontakt z client architects.
• Współpraca z Databricks Lead i Governance oraz z zespołami klienta każdego dnia.
• Prowadzenie projektów, wsparcie przy bidach, mentoring inżynierów i utrzymanie certyfikacji.
• Zarządzanie kosztami platformy, tuning wydajności, monitorowanie i orkestracja zadań.
• Wsparcie warstwy governance: jakość danych, lineage, Unity Catalog.
• W roli senior: planowanie zakresów, wspieranie bidów i rozwój zespołu.

Skills

Databricks

Spark

PySpark

Scala

SQL

Python

Delta Lake

Lakeflow

Databricks SQL

Azure

AWS

GCP

Git

CI/CD

Terraform

Databricks Asset Bundles

Unity Catalog

Tools

Terraform

Databricks Asset Bundles

Job description

Nasze wymagania:

Engineer: 3+ years in data engineering, with at least one year hands-on on Databricks in production. Senior Engineer: 5+ years, at least three on Databricks, with hands-on work in at least three of: platform setup, batch and streaming pipelines, data modelling, migration, CI/CD and infrastructure as code, performance and cost tuning, and a track record of leading the technical side of a workstream. Strong Spark (PySpark or Scala) and SQL; Python; Delta Lake; Lakeflow pipelines and jobs; Databricks SQL. Working knowledge of one cloud (Azure preferred; AWS or GCP): storage, identity and networking as they apply to Databricks. Git and CI/CD; experience with Databricks Asset Bundles or Terraform (both, at senior level). Databricks Certified Data Engineer Associate held or in progress (Professional, at senior level). Unity Catalog basics: catalogs, permissions, lineage; interest in data governance is a plus. Clear communication with client data teams; fluent English; at senior level, the ability to present a technical design to client architects. Experience in financial services, energy, pharma or public sector is a plus.

O projekcie:

You will build lakehouse platforms on Databricks for our clients, end to end: workspace and platform setup, ingestion, pipelines, data model, data products for analytics and machine learning, and the operations that keep it all running. Our engineers are not split into platform people and pipeline people; you cover the whole stack and grow across it. As an engineer you own pipelines, data products and parts of the platform inside a workstream. As a senior engineer you own the technical design of a workstream or of a whole platform, review the work of the other engineers and are the person the client's architects talk to. You will work with the Databricks Lead on architecture, with the Senior Databricks Engineer - Governance when governance is part of the engagement, and with the client's data teams every day. We hire at both levels on this description; what changes is the share of design and review versus building.

Zakres obowiązków:

• Set up and configure Databricks platforms: workspaces, compute policies, Unity Catalog basics, networking and identity with the platform team; at senior level, own the platform design and its cost model.
• Build ingestion and pipelines: Lakeflow Connect and Auto Loader, Lakeflow pipelines in batch and streaming, Spark and SQL transformations, Delta Lake tables in a medallion model; at senior level, set the patterns the other engineers follow.
• Model data for consumption: data products, Databricks SQL and AI/BI dashboards, feature tables and model serving with MLflow; at senior level, decide the modelling approach with the client.
• Migrate workloads from legacy warehouses, Hadoop or notebooks to Databricks; refactor what cannot be migrated as it is.
• Engineer like software: Git, CI/CD with Databricks Asset Bundles, infrastructure as code with Terraform, automated tests, code review; at senior level, run the reviews and own the release process.
• Run and optimise: monitoring, job orchestration, performance tuning, cost control (cluster sizing, serverless, Photon, query tuning).
• Support the governance layer where the engagement includes it: data quality expectations in pipelines, lineage, access requests in Unity Catalog, together with the governance engineer.
• At senior level: estimate and plan workstreams, support bids with the Databricks Lead, coach engineers and keep certifications current across the team.

Oferujemy:

• Practical benefits: yearly financial bonus, private medical care with Medicover with additional packages (e.g., dental, senior care, oncology) available on preferential terms, life insurance and access to NAIS benefit platform.
• Access to over 70 training tracks with certification opportunities (e.g., GenAI, Excel, Business Analysis, Project Management) on our NEXT platform. Dive into a world of knowledge with free access to Education First languages platform, TED Talks and Udemy Business materials and trainings.
• Cutting-Edge Technology: Position yourself at the forefront of IT innovation, working with the latest technologies and platforms. Capgemini partners with top global enterprises, including 145 Fortune 500 companies.
• Enjoy hybrid working model that fits your life - after completing onboarding, connect work from a modern office with ergonomic work from home, thanks to home office package (including laptop, monitor, and chair). Ask your recruiter about the details.