Intermediate - Senior Data Engineer (Databricks)

aica consultancy

Deutschland

On-site

EUR 75,000 - 110,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

AICA Consultancy is seeking a Data Engineer in Germany with Databricks certification to design, develop, and optimize scalable data pipelines and workflows. You will work with stakeholders to ensure data reliability, performance, and alignment with business requirements.

Responsibilities include building ETL/ELT pipelines on Databricks, implementing Delta Lake optimizations, and enabling real-time streaming.

Qualifications

  • Databricks certified is required.
  • Proficiency in Spark and Delta Lake.
  • Strong Python and SQL skills for ETL/ELT development.

Responsibilities

  • Build and optimize scalable data pipelines on Databricks.
  • Develop real-time streaming solutions with Databricks Structured Streaming.
  • Collaborate with stakeholders to ensure data reliability and performance.
  • Document pipelines, workflows, and technical solutions.

Skills

Databricks
Spark
Delta Lake
Python
SQL
ETL/ELT development
AWS
Azure
GCP
Unity CatLog

Tools

Unity CatLog

Job description

We are looking for a Data Engineer who is certified in Databricks (required) to join our team. In this role you will be designing, developing, and optimizing scalable data pipelines and workflows on Databricks. The engineer will work closely with stakeholders to make certain data reliability, performance, and alignment with business requirements.

Scope of Work
Data Pipeline Development
  • Building efficient ETL/ELT pipelines using Databricks and Delta Lake for structured, semi-structured, and unstructured data.
  • Transforming raw data into consumable datasets for analytics and machine learning.
Data Optimization
  • Improving performance by implementing best practices like partitioning, caching, and Delta Lake optimizations.
  • Resolving bottlenecks and ensuring scalability.
Data Integration
  • Integrating data from various sources such as APIs, databases, and cloud storage systems (e.g., AWS S3, Azure Data Lake).
Real-Time Streaming
  • Designing and deploying real-time data streaming solutions using Databricks Structured Streaming.
Data Quality and Governance
  • Implementing data validation, schema enforcement, and monitoring to ensure high-quality data delivery.
  • Using Unity CatLog to manage metadata, access permissions, and data lineage.
Collaboration and Documentation
  • Collaborating with data analysts, data scientists, and other stakeholders to meet business needs.
  • Documenting pipelines, workflows, and technical solutions.
Responsibilities
  • Fully functional and documented data pipelines.
  • Optimized and scalable data workflows on Databricks.
  • Real-time streaming solutions integrated with downstream systems.
  • Detailed documentation for implemented solutions and best practices.
Skills and Qualifications
  • Proficiency in Databricks (certified), Spark, and Delta Lake.
  • Strong experience with Python, SQL, and ETL/ELT development.
  • Familiarity with real-time data processing and streaming.
  • Knowledge of cloud platforms (e.g., AWS, Azure, GCP).
  • Experience with data governance and tools like Unity CatLog.
Assumptions
  • Access to necessary datasets and cloud infrastructure will be provided.
  • Timely input and feedback from stakeholders.
Success Metrics
  • Data pipelines deliver accurate and consistent data.
  • Workflows meet performance benchmarks.
  • Real-time streaming solutions operate with minimal latency.
  • Stakeholders are satisfied with the quality and usability of the solutions.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Y-Axis • Berlin

Hybrid
EUR 90,000 - 150,000
Senior Staff Software Engineer - Delta
Senior Staff Software Engineer - Delta

Neon • Berlin

On-site
EUR 150,000 - 190,000
Senior Staff Software Engineer - Delta
Senior Staff Software Engineer - Delta

Databricks Inc. • Berlin

On-site
EUR 90,000 - 120,000
Comprehensive benefits
Inclusive culture
Data Solution Architect
Data Solution Architect

LTM • München

Hybrid
EUR 110,000 - 150,000
Data Engineer
Data Engineer

MediaRadar • Germany

Remote
EUR 90,000 - 120,000
Solutions Architect
Solutions Architect

Neon • München

On-site
EUR 70,000 - 90,000
Comprehensive benefits
Commitment to diversity and inclusion
Solutions Architect
Solutions Architect

Databricks Inc. • München

On-site
EUR 80,000 - 100,000
Staff Forward Deployed Engineer
Staff Forward Deployed Engineer

Neon • Berlin

On-site
EUR 120,000 - 170,000
Staff Data Engineer (all genders)
Staff Data Engineer (all genders)

Eurowings Digital • Köln

On-site
EUR 70,000 - 90,000
Staff Forward Deployed Engineer
Staff Forward Deployed Engineer

FDE Times • München

Hybrid
EUR 110,000 - 140,000