Senior Data Engineer (Databricks, Azure)

Acuity Analytics; Porto; On-site; EUR 50,000 - 75,000; Full time

Job summary

Acuity Analytics is seeking a skilled Data Engineer in Porto to design, build, and manage scalable data solutions in a cloud-native environment. You will focus on automation, performance, and collaboration across teams to deliver end-to-end data pipelines using Databricks and Azure.

Responsibilities include creating Databricks Workflows, deploying PyFunc models, and ensuring secure configuration with Azure Key Vault, all while integrating Unity Catalog along the data lakehouse pattern.

Qualifications

• Experience with Databricks, Azure Cloud, and data orchestration.
• Design, build scalable data solutions in a cloud-native environment.
• Strong Python skills for data engineering and automation.

Responsibilities

• Build and manage Databricks Workflows for large-scale data orchestration.
• Develop and deploy PyFunc models into production pipelines.
• Implement secure secrets and configuration management using Azure Key Vault.
• Design and optimize ETL/ELT pipelines following best practices.
• Govern and manage data assets using Unity Catalog and Data Lakehouse patterns.
• Implement messaging and event-driven workflows with Azure Service Bus.
• Collaborate with architects, analysts, and data scientists to deliver end-to-end solutions.

Skills

Python

Production-ready code

CI/CD

Tools

Databricks

Azure

Unity Catalog

Job description

If you like providing efficient, effective solutions to complex challenges, join us at Acuity! Be at the heart of helping achieve extraordinary things with software and data for our customers!

About Us

Ascent has recently been acquired by Acuity Analytics. This is both a significant milestone for us and a tremendous opportunity for you. Acuity Analytics is a business with a strong global reputation, an impressive client base and ambitious growth plans. We delivers deep insights and domain-led digital transformation to high-growth and heavily regulated organisations. To our customers, we bring a partnership that provides the talent, technology and capability to enhance performance and operational efficiency.

About the role

We are looking for a skilled Data Engineer with strong experience in Databricks, Azure Cloud, and data orchestration to join our team. You will be responsible for designing, building, and managing scalable data solutions in a cloud-native environment, with a focus on automation, performance, and collaboration.

Key Responsibilities

• Build and manage Databricks Workflows for large-scale data orchestration.
• Develop and deploy PyFunc models and integrate them into production pipelines.
• Implement secure secrets and configuration management using Azure Key Vault.
• Design and optimize ETL / ELT pipelines in line with best practices.
• Govern and manage data assets using Unity Catalog and Data Lakehouse patterns.
• Implement messaging and event-driven workflows with Azure Service Bus.
• Collaborate with architects, analysts, and data scientists to deliver end-to-end solutions.

Skills Required

Programming

• Strong Python skills for data engineering, automation, and integration
• Experience writing modular, production-ready code
• CI/CD pipelines for Databricks and Azure

Databases

• Strong knowledge of SQL (including query optimization and data modeling)
• API Management (APIM)

Nice to Haves

AI & Cognitive Services

• Experience with Microsoft Cognitive Services (OCR, Layout, Form Recognizer, etc.)
• Integration of cognitive services into data pipelines
• Familiarity with event-driven and microservices architecture patterns
• Experience with monitoring tools like Datadog, Application Insights, or similar
• Agile team collaboration
• DevOps practices

People are at the Heart of our Business. By investing in people, we achieve exceptional results for our clients and create new opportunities for our teams to thrive. Check out this page for more details.