Data Engineer (Data Bricks)

DataEdge SRL; Frankfurt; Vor Ort; EUR 85.000 - 110.000; Vollzeit

Zusammenfassung

DataEdge SRL is seeking a hands-on Data Engineer in Frankfurt to design and implement scalable data pipelines using Databricks and Spark. You will work with data scientists and stakeholders to deliver production-grade data platforms for AI/ML and RAG use cases.

The role requires strong PySpark development, data modeling expertise, and experience handling large-scale, unstructured data across cloud environments. EU location is mandatory.

Qualifikationen

• Strong hands-on experience with Databricks in production environments.
• In-depth knowledge of Apache Spark and PySpark, including performance tuning.
• Strong Data Engineering background with production-grade data pipelines.
• Solid understanding of data modeling and modern data architectures.
• Proven experience processing large-scale datasets.
• Experience working with unstructured and semi-structured data.
• Practical experience preparing and transforming data for AI/ML and RAG use cases.
• Strong Python skills for data engineering and PySpark development.
• Experience with data ingestion, transformation, orchestration, and pipeline automation.
• Candidate must be located within the EU.

Aufgaben

• Design, develop, and optimize data engineering pipelines and data processing solutions using Databricks and Apache Spark.
• Build scalable and reliable data pipelines using PySpark and related Spark technologies.
• Work with large and complex datasets across structured, semi-structured, and unstructured data sources.
• Design and implement effective data models to support analytics, AI/ML, and downstream data consumption.
• Process and transform unstructured and semi-structured data for AI-driven use cases, including RAG.
• Develop data ingestion, transformation, cleansing, and enrichment workflows.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and reliability.
• Work closely with data scientists, ML/AI engineers, architects, and business stakeholders to deliver production-ready data solutions.
• Apply strong engineering practices around data quality, testing, monitoring, and operational reliability.

Kenntnisse

Databricks

Apache Spark

PySpark

Data Engineering

Data Modeling

Large-Scale Datasets

AI/ML Data Prep

Unstructured Data

Python

Data Ingestion

Tools

Delta Lake

Azure/AWS/GCP

Jobbeschreibung

Data Engineer

Location: Frankfurt, Germany / EU
Start: ASAP

We are looking for a strong, hands-on Data Engineer with deep Databricks and Apache Spark experience to support a Frankfurt-based banking client.

The ideal candidate will have extensive experience designing and implementing scalable data engineering solutions using Databricks, Spark/PySpark, with a strong understanding of data modeling and modern data architectures. Experience working with unstructured and semi-structured data for AI/ML and RAG use cases is highly desirable.

Key Responsibilities:

• Design, develop, and optimize data engineering pipelines and data processing solutions using Databricks and Apache Spark.
• Build scalable and reliable data pipelines using PySpark and related Spark technologies.
• Work with large and complex datasets across structured, semi-structured, and unstructured data sources.
• Design and implement effective data models to support analytics, AI/ML, and downstream data consumption.
• Process and transform unstructured and semi-structured data for AI-driven use cases, including RAG (Retrieval-Augmented Generation).
• Develop data ingestion, transformation, cleansing, and enrichment workflows.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and reliability.
• Work closely with data scientists, ML/AI engineers, architects, and business stakeholders to deliver production-ready data solutions.
• Apply strong engineering practices around data quality, testing, monitoring, and operational reliability.
• Contribute to the design and evolution of modern cloud-based data platforms.

Must-Have Requirements:

• Strong hands-on experience with Databricks in production environments.
• In-depth knowledge of Apache Spark and PySpark, including performance tuning and optimization.
• Strong Data Engineering background, with experience building production-grade data pipelines.
• Solid understanding of data modeling, data structures, and modern data architectures.
• Proven experience processing large-scale datasets.
• Experience working with unstructured and semi-structured data.
• Practical experience preparing and transforming data for AI/ML and RAG use cases.
• Strong Python skills, particularly for data engineering and PySpark development.
• Experience with data ingestion, transformation, orchestration, and pipeline automation.
• Ability to work independently in a fast-paced banking/enterprise environment.
• Candidate must be located within the EU.

Nice-to-Have:

• Experience with Generative AI / LLM / RAG architectures.
• Knowledge of vector search, embeddings, chunking, and document-processing pipelines.
• Experience with Delta Lake / Delta tables and modern lakehouse architectures.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Experience in banking or other regulated financial-services environments.
• Knowledge of data governance, security, lineage, and compliance requirements.
• Experience with CI/CD and DevOps practices for data platforms.