Lead Data Engineer (m/f/d) for the BMP Migration Tool

softwareXperts GmbH; München; A distancia; EUR 90.000 - 130.000; Jornada completa

Descripción de la vacante

softwareXperts GmbH is seeking a Lead Data Engineer (m/f/d) for the BMP Migration Tool. This role offers full remote work with 2–3 days per year on-site in Germany.

You will provide hands-on technical leadership, drive data profiling, define mappings, and build pipelines using Databricks, PySpark, Spark SQL, and Delta Lake. The ideal candidate has strong experience in data migrations, data quality monitoring, CI/CD, and excellent English communication in a hybrid setup.

Formación

• Extensive hands-on experience delivering data migrations and building production-grade data pipelines.
• Deep Databricks expertise including Unity Catalog and Delta Lake.
• Strong PySpark and Spark SQL skills with data profiling and reconciliation.
• Experience with Databricks Data Quality Monitoring and Lakebridge.
• Proven technical leadership and ability to coordinate delivery.

Responsabilidades

• Provide hands-on technical leadership for the BMP data migration.
• Lead profiling and assessment of source data.
• Define and implement data mappings and transformation rules.
• Develop and maintain data pipelines using Databricks, PySpark, Spark SQL and Delta Lake.
• Establish and oversee data quality checks and monitoring.
• Lead source-to-target reconciliation and resolve data discrepancies.
• Guide testing, CI/CD deployment and code quality practices.
• Prepare runbooks and support cutover activities

Conocimientos

Databricks

PySpark

Spark SQL

Data migration

Data pipelines

Data profiling

CI/CD

Data quality

Leadership

Unity Catalog

Lakeflow

Herramientas

Delta Lake

Lakeflow Connect

Lakeflow Declarative Pipelines

Lakeflow Jobs

Lakebridge

DQX

Databricks Asset Bundles

CI/CD

Descripción del empleo

Lead Data Engineer (m/f/d) for the BMP Migration Tool

ID: 10352

Our client is a service company that operates within an international corporate group.

Full remote

2–3 days per year on site

Must haves

• Databricks
• Lakeflow
• SQL

Nice to have

• Unity Catalog
• PySpark

Your Tasks

• Provide hands-on technical leadership for the BMP data migration.
• Lead the profiling and assessment of source data.
• Define and implement data mappings and transformation rules.
• Develop and maintain data pipelines using Databricks, PySpark, Spark SQL and Delta Lake.
• Establish and oversee data quality checks and monitoring.
• Lead source-to-target reconciliation and resolve data discrepancies.
• Guide testing, CI/CD deployment and code quality practices.
• Prepare runbooks and support cutover activities

Requirements

• Strong hands-on experience delivering data migrations and building production-grade data pipelines.
• Extensive practical experience with Databricks , including Unity Catalog, Delta Lake, Lakeflow Connect, Lakeflow Declarative Pipelines and Lakeflow Jobs.
• Strong PySpark and Spark SQL skills.
• Experience with data profiling, source-to-target mapping, reconciliation and data quality checks.
• Familiarity with DQX , Databricks Data Quality Monitoring and Lakebridge.
• Experience with Databricks Asset Bundles, CI/CD and automated testing.
• Ability to prepare runbooks, troubleshoot migration issues and support cutover activities.
• Proven technical leadership, including guiding implementation and coordinating delivery.
• Required knowledge of: Databricks, Unity Catalog, Lakeflow Connect, Lakeflow Declarative Pipelines, Lakeflow Jobs, Delta Lake, PySpark, Spark SQL, DQX, Databricks Data Quality Monitoring, Lakebridge, Databricks Asset Bundles, CI/CD, Lakehouse Federation and AI/BI Dashboards.
• Professional English communication skills; willingness to work in a hybrid setup.
• Availability to start as soon as possible.