Senior Data Engineer - Azure Databricks Lakehouse Expert

Softlanding; Portugal; Remote; EUR 60,000 - 90,000; Full time

Job summary

Softlanding is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines on an Azure Databricks-based Lakehouse platform. You will ensure data quality, security, and governance while delivering business-ready data assets.

You will collaborate with Data Architects, BI teams, and stakeholders, participate in Agile ceremonies, and contribute to reusable data engineering components and deployment automation.

Qualifications

• Minimum 4+ years of experience in Data Engineering.
• Minimum 2+ years of hands-on development experience with Azure Databricks.
• Strong proficiency in PySpark and SQL.
• Practical experience with Azure Data Platform services.
• Proven experience designing, developing, and maintaining ETL/ELT pipelines.
• Good understanding of Lakehouse concepts and Medallion Architecture (Bronze, Silver, Gold).
• Experience integrating data from multiple enterprise systems, databases, APIs, and external sources.
• Working knowledge of Git, version control, and CI/CD practices.
• Understanding of data quality, governance, security, and access management principles.
• Mandatory: Valid Visual Studio Professional license and ability to work within licensed development environment.

Responsibilities

• Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Databricks and PySpark.
• Build and manage Bronze, Silver, and Gold layers following Medallion Architecture principles.
• Develop robust ETL/ELT workflows to process structured and semi-structured data.
• Integrate data from multiple enterprise systems, APIs, databases, and external data sources.
• Implement data transformation, validation, and processing logic using PySpark and SQL.
• Collaborate with Data Architects to translate designs into production-ready pipelines.
• Work with BI/Analytics teams to deliver trusted datasets for reporting and self-service analytics.
• Participate in Agile ceremonies, code reviews, and release activities.
• Use Git-based version control and CI/CD practices to manage code quality and deployments.
• Maintain technical documentation for data pipelines and platform components.

Skills

Data Engineering

Azure Databricks

PySpark

SQL

ETL/ELT pipelines

Data quality

Data governance

CI/CD

Git

Security

Tools

Git

CI/CD

Visual Studio

Job description

Softlanding is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines on an Azure Databricks-based Lakehouse platform. You will ensure data quality, security, and governance while delivering business-ready data assets.

You will collaborate with Data Architects, BI teams, and stakeholders, participate in Agile ceremonies, and contribute to reusable data engineering components and deployment automation.