Databricks Lakehouse Data Engineer - Pipelines & Governance

Jobtailor; Lisboa; Presencial; EUR 50 000 - 70 000; Tempo integral

Resumo da oferta

Jobtailor in Lisbon is seeking a data engineer to build and maintain scalable data pipelines on the Databricks Lakehouse Platform using Spark and PySpark. You will ingest data from databases, files and APIs, model bronze/silver/gold transforms, and ensure data quality and governance.

You will optimize performance and costs, apply security and Unity Catalog governance, contribute to CI/CD for data assets, and collaborate with data scientists and analysts to deliver reliable datasets for analytics

Qualificações

• Bachelor's or Master's degree in CS, IS, Engineering, or related field.
• 2+ years of professional experience in data engineering or software engineering with a strong data component.
• Solid Python skills in production environments.
• Hands-on experience with Spark/PySpark for large-scale batch or streaming data processing.

Responsabilidades

• Build and maintain scalable data processing pipelines on the Databricks Lakehouse Platform using Apache Spark and PySpark.
• Ingest and integrate datasets from databases, file systems, and APIs into the Lakehouse using Auto Loader, CDC, and batch or streaming ingestion patterns.
• Model and implement bronze, silver, and gold transformation layers using Delta Lake and Databricks SQL.
• Ensure data integrity, consistency, and quality through validation and monitoring using Delta Live Tables expectations and Lakehouse monitoring capabilities.
• Tune pipelines for performance and cost through Spark job optimization, Delta table maintenance, and cluster and compute configuration.
• Apply data security, access control, and privacy practices using Unity Catalog for governance, permissions, and lineage.
• Contribute to CI/CD and deployment automation for data assets.
• Support migration initiatives from on-premise Cloudera environments to the Databricks Lakehouse.
• Collaborate with data scientists and analysts to deliver reliable datasets for analytics and machine learning workflows.
• Occasionally support lightweight Python services and APIs exposing data to downstream consumers.

Conhecimentos

Python

SQL

Apache Spark

PySpark

Data Modeling

Delta Lake

Delta Live Tables

Structured Streaming

Data Ingestion

CI/CD

Testing

Version Control

Collaboration

Communication

Formação académica

Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field

Ferramentas

Databricks

Airflow

Kafka

Terraform

Descrição da oferta de emprego

Jobtailor in Lisbon is seeking a data engineer to build and maintain scalable data pipelines on the Databricks Lakehouse Platform using Spark and PySpark. You will ingest data from databases, files and APIs, model bronze/silver/gold transforms, and ensure data quality and governance.

You will optimize performance and costs, apply security and Unity Catalog governance, contribute to CI/CD for data assets, and collaborate with data scientists and analysts to deliver reliable datasets for analytics