Senior Data Engineer

Softlanding; Portugal; Remote; EUR 60,000 - 90,000; Full time

Job summary

Softlanding is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines on an Azure Databricks-based Lakehouse platform. You will ensure data quality, security, and governance while delivering business-ready data assets.

You will collaborate with Data Architects, BI teams, and stakeholders, participate in Agile ceremonies, and contribute to reusable data engineering components and deployment automation.

Qualifications

• Minimum 4+ years of experience in Data Engineering.
• Minimum 2+ years of hands-on development experience with Azure Databricks.
• Strong proficiency in PySpark and SQL.
• Practical experience with Azure Data Platform services.
• Proven experience designing, developing, and maintaining ETL/ELT pipelines.
• Good understanding of Lakehouse concepts and Medallion Architecture (Bronze, Silver, Gold).
• Experience integrating data from multiple enterprise systems, databases, APIs, and external sources.
• Working knowledge of Git, version control, and CI/CD practices.
• Understanding of data quality, governance, security, and access management principles.
• Mandatory: Valid Visual Studio Professional license and ability to work within licensed development environment.

Responsibilities

• Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Databricks and PySpark.
• Build and manage Bronze, Silver, and Gold layers following Medallion Architecture principles.
• Develop robust ETL/ELT workflows to process structured and semi-structured data.
• Integrate data from multiple enterprise systems, APIs, databases, and external data sources.
• Implement data transformation, validation, and processing logic using PySpark and SQL.
• Collaborate with Data Architects to translate designs into production-ready pipelines.
• Work with BI/Analytics teams to deliver trusted datasets for reporting and self-service analytics.
• Participate in Agile ceremonies, code reviews, and release activities.
• Use Git-based version control and CI/CD practices to manage code quality and deployments.
• Maintain technical documentation for data pipelines and platform components.

Skills

Data Engineering

Azure Databricks

PySpark

SQL

ETL/ELT pipelines

Data quality

Data governance

CI/CD

Git

Security

Tools

Git

CI/CD

Visual Studio

Job description

We are seeking an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines and data products on an Azure Databricks-based Lakehouse platform. The successful candidate will be responsible for implementing data ingestion, transformation, and consumption layers while ensuring high standards of performance, reliability, security, and data quality.

The candidate will collaborate closely with Data Architects, BI and Analytics teams, and business stakeholders to deliver trusted, scalable, and business-ready data assets aligned with enterprise data architecture standards.

Key Responsibilities

1. Data Pipeline Development

• Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Databricks and PySpark.
• Build and manage Bronze, Silver, and Gold layers following Medallion Architecture principles.
• Develop robust ETL/ELT workflows to process structured and semi-structured data.
• Integrate data from multiple enterprise systems, APIs, databases, and external data sources.
• Implement data transformation, validation, and processing logic using PySpark and SQL.

2. Data Platform Development and Optimization

• Implement data solutions in accordance with architecture standards and guidelines defined by the Data Architect.
• Develop reusable data engineering frameworks, components, and pipeline templates.
• Optimize pipeline performance, scalability, reliability, and compute costs.
• Apply best practices for data processing, incremental loading, error handling, and recovery.
• Support the development and maintenance of enterprise Lakehouse data products.

3. Data Quality, Governance, and Security

• Implement data quality checks, validation rules, and monitoring mechanisms.
• Support data lineage, cataloging, metadata management, and governance initiatives.
• Ensure compliance with enterprise security, access control, and data protection standards.
• Follow best practices for data access management and secure data processing.

4. Collaboration and Delivery

• Work closely with Data Architects to translate solution designs into production-ready data pipelines.
• Collaborate with BI developers, Analytics teams, and business stakeholders to deliver trusted datasets for reporting and self-service analytics.
• Participate in Agile ceremonies, sprint planning, code reviews, and release activities.
• Use Git-based version control and CI/CD practices to manage code quality and deployments.
• Maintain technical documentation for data pipelines, transformations, and platform components.

Mandatory Skills and Experience

• Minimum 4+ years of experience in Data Engineering.
• Minimum 2+ years of hands-on development experience with Azure Databricks.
• Strong proficiency in PySpark and SQL.
• Practical experience with Azure Data Platform services.
• Proven experience designing, developing, and maintaining ETL/ELT pipelines.
• Good understanding of Lakehouse concepts and Medallion Architecture (Bronze, Silver, and Gold layers).
• Experience integrating data from multiple enterprise systems, databases, APIs, and external sources.
• Working knowledge of Git, version control, and CI/CD practices.
• Understanding of data quality, data governance, security, and access management principles.
• Mandatory: Valid Visual Studio Professional license and the ability to work within the organization's licensed development environment.

Preferred Skills

• Experience with Delta Live Tables (DLT).
• Hands-on experience with Unity Catalog.
• Exposure to Microsoft Fabric.
• Experience working on Data Hub, Enterprise Data Platform, or large-scale data modernization programs.
• Familiarity with automated data quality frameworks, pipeline monitoring, and production support.
• Experience implementing reusable data engineering frameworks and deployment automation.

Key Competencies

• Strong analytical and problem-solving skills.
• Ability to translate technical architecture into scalable data engineering solutions.
• Effective communication and collaboration across technical and business teams.
• Commitment to code quality, maintainability, performance, and data reliability.
• Ability to work effectively in an Agile delivery environment.

Expected Outcomes

• Reliable, scalable, and cost-efficient data pipelines.
• Well-structured and governed Bronze, Silver, and Gold data layers.
• High-quality, trusted datasets that support enterprise reporting and analytics.
• Reusable data engineering components aligned with enterprise architecture standards.
• Secure, maintainable, and production-ready data solutions.