Databricks Data Engineer — Pipelines, dbt & Snowflake

Jobtailor; Barcelona; Presencial; EUR 70.000 - 100.000; Jornada completa

Descripción de la vacante

Jobtailor is seeking a data engineer to own and operate production data pipelines on Databricks, with strong PySpark/SQL expertise. You will migrate legacy Hive Metastore to Unity Catalog and maintain Iceberg sharing with Snowflake, while building dbt models and Airflow DAGs.

You will work closely with client stakeholders and the team lead to optimize performance and cost, ensuring data quality and reliable incident handling in a fast-paced environment.

Formación

• 3+ years operating production data pipelines.
• Proficient in PySpark and SQL to read, debug, and modify existing pipelines.
• Experience with Delta Lake MERGE/upsert, table properties, and partitioning.
• Knowledge of Databricks Workflows, cluster config, and job troubleshooting.
• Familiarity with Unity Catalog catalogs, schemas, grants, lineage, and metastore model.
• Experience with Snowflake warehouses, roles and grants, and general operating model.
• Ability to read and optimize Snowflake queries and manage compute costs.
• Experience with dbt models, sources, tests, and incremental materializations.
• Understanding of dbt project structure and deployment workflow.
• Experience building and maintaining Airflow DAGs and orchestration.
• Handling retries, backfills, and idempotent task design.
• Strong SQL including window functions, joins, and reading transformation logic.
• Python scripting for automation and API integration.
• Incremental loading patterns, late-arriving data handling, and reprocessing.
• Basic AWS S3 and IAM knowledge.
• Basic understanding of Redshift and its role in architecture.
• Fluent English.
• Self-directed and able to navigate unfamiliar codebases without onboarding.
• Ability to explain production incidents to non-technical stakeholders and provide ETAs.

Responsabilidades

• Keep Databricks pipelines running for ingestion, transformation, and delivery to downstream consumers.
• Diagnose and resolve pipeline failures and data quality issues.
• Reverse-engineer and document existing transformation logic and business rules.
• Migrate legacy Hive Metastore tables to Unity Catalog.
• Maintain Iceberg-enabled table sharing between Databricks and Snowflake.
• Build and test dbt models with incremental materializations and data tests.
• Develop and maintain Airflow DAGs for orchestration.
• Validate migrated pipelines against Databricks outputs.
• Contribute to Snowflake modeling, performance, and cost decisions.
• Collaborate with client stakeholders on technical topics alongside the team lead.

Conocimientos

SQL

PySpark

Dbt

Airflow

Delta Lake

Unity Catalog

Snowflake

AWS S3

Incremental Loading

Data Quality

Herramientas

Databricks

Snowflake

Airflow

Hive Metastore

Redshift

Descripción del empleo

Jobtailor is seeking a data engineer to own and operate production data pipelines on Databricks, with strong PySpark/SQL expertise. You will migrate legacy Hive Metastore to Unity Catalog and maintain Iceberg sharing with Snowflake, while building dbt models and Airflow DAGs.

You will work closely with client stakeholders and the team lead to optimize performance and cost, ensuring data quality and reliable incident handling in a fast-paced environment.