Data Engineer - Databricks (gn)

BLACKBULL INTERNATIONAL GmbH; Frankfurt; Vor Ort; EUR 75.000 - 110.000; Vollzeit

Zusammenfassung

BLACKBULL INTERNATIONAL GmbH, a globally active asset management firm, is seeking a Data Engineer - Databricks to design, build, and operate data products on a Databricks-based platform. You will create reusable pipeline components and enable internal users through guidance, while ensuring security, privacy and regulatory compliance.

You will architect data workflows, govern data assets across maturity tiers, and collaborate with AI/ML teams to support retrieval-augmented generation and

Qualifikationen

• At least 3 years of hands-on experience building large-scale data pipelines and data products on Databricks.
• Experience in regulated environments, ideally in Financial Services.
• Proficient in Databricks, Spark, Python and SQL with CI/CD and IaC tooling (Terraform) on Azure.
• Strong data engineering principles for AI/ML, data modelling, quality and reproducibility.
• Knowledge of Lakehouse, layered data, data product thinking in governed enterprise.
• Degree in computer science or a comparable discipline.
• Strong documentation and ability to transfer knowledge.

Aufgaben

• Develop and maintain a library of modular, reusable pipeline components for AI/ML-ready datasets.
• Architect and run data workflows on Databricks to produce clean data assets for AI, ML and reporting.
• Govern lifecycle of layered data assets with quality and timeliness standards.
• Build and maintain transformation workflows deriving predictive attributes and a central attribute repository.
• Collaborate with AI/ML engineers to establish data supply chains for retrieval-augmented generation.
• Embed governance controls into platform design to meet policies and regulatory requirements.

Kenntnisse

Data engineering expertise

Analytical mindset

Documentation and knowledge transfer

Ausbildung

Bachelor's degree in computer science

Tools

Databricks

Spark

Python

SQL

CI/CD

Terraform

Azure

Jobbeschreibung

For our client, a globally active asset management firm with a strong presence across institutional and private client segments, we are currently looking for a Data Engineer - Databricks (gn) to strengthen the team.

Purpose of job

The role is responsible for designing, building and operating data products and reusable data preparation components on a Databricks-based Data and AI Platform, while acting as a technical enabler for internal platform users through expert guidance and advanced-level support. The position ensures adherence to security, privacy and regulatory requirements via a compliance-by-design approach, maintains alignment with established best practices for data pipelines and data quality, and drives continuous platform improvement through the integration of new features and standardized, reusable pipelines.

Tasks

• Develop and maintain a library of modular, reusable pipeline components — covering data intake, processing, verification and enrichment — to produce consistently structured, AI/ML-ready datasets from both structured and unstructured sources.
• Architect and run dependable data workflows on Databricks, pulling from a wide range of internal and external origins to produce clean, validated data assets available for AI, ML and reporting purposes.
• Govern the lifecycle of layered data assets across maturity tiers, upholding quality and timeliness standards while maintaining purpose‑specific, analytics‑and model‑ready output datasets.
• Build and maintain transformation workflows that derive meaningful predictive attributes from raw data, alongside a centrally managed attribute repository with clear versioning, ownership and service‑level commitments.
• Work alongside AI and ML engineers to establish and maintain data supply chains for retrieval‑augmented generation systems, covering content segmentation, vector representation updates and index synchronization.
• Embed governance controls — covering permissions, traceability, data lifecycle management, encryption and audit trails — into platform design to meet both internal policies and external regulatory requirements.

Required:

• At least 3 years of hands‑on experience designing and running large‑scale data pipelines and data products on Databricks — batch and/or streaming — preferably in regulated or governance‑heavy environments (ideally in the Financial Services industry)
• Advanced proficiency in Databricks, Spark, Python and SQL, complemented by sound software engineering practices such as CI/CD workflows and infrastructure‑as‑code tooling (e.g., Terraform) on Azure.
• Solid grasp of data engineering principles relevant to AI/ML contexts, including data modelling, quality assurance, feature collaboration and reproducibility, as well as an understanding of how data characteristics influence model behaviour.
• Strong command of modern data architecture patterns — including Lakehouse principles, layered data organisation and data product thinking — and the ability to put them into practice within a governed enterprise setting.
• Degree in computer science or a comparable discipline.
• Strong analytical mindset paired with structured planning, clear documentation and the ability to effectively transfer knowledge to colleagues.

Preferred:

• Professional background in the financial sector, ideally within asset management, combined with international work experience and relevant Databricks certifications.