Senior Data Engineer (Databricks)

DataOnline Corp.; Porto; Presencial; EUR 55 000 - 90 000; Tempo integral

Resumo da oferta

Anova is seeking a Data Engineer (Databricks) to join the R&D Software team in Porto on a hybrid basis. Most work can be done remotely, with occasional in-office time for collaboration.

You will deliver end-to-end Databricks ETL, translate business goals into data solutions, and monitor pipelines in production while ensuring data quality and governance. Leverage AI-assisted coding tools and maintain clear data lineage.

Qualificações

• Bachelor's degree or equivalent in a quantitative field
• 5+ years in data engineering with Databricks experience
• Strong PySpark and Delta Lake understanding
• Solid SQL skills including window functions and joins
• Experience with streaming or incremental ingestion and data modelling
• Familiar with CI/CD for Databricks workflows
• Data quality testing and governance awareness
• Proficient in written and spoken English

Responsabilidades

• Deliver end-to-end Databricks ETL projects from requirements to production
• Translate business goals into data solutions for stakeholders
• Own and monitor data pipelines and ensure reliability
• Build and maintain the One Anova data stream and BI assets
• Work with real-time telemetry from IoT sensors
• Ensure data lineage, governance, and master data quality

Conhecimentos

Databricks

PySpark

Delta Lake

Advanced SQL

Streaming data

CI/CD pipelines

Data modelling

Data quality testing

Code reviews

CI/CD

Formação académica

Bachelor's degree in CS / Data / related field

Ferramentas

Git

Terraform

Descrição da oferta de emprego

Join us on the R&D Software team as a Data Engineer (Databricks), and help shape the future of safer, more efficient, and more reliable operations across the globe. Start your journey with Anova today!

Where you’ll work: This is a hybrid role based out of our Porto office. In practice, most of your work can be done remotely, with occasional in-office time in Porto for team collaboration — a flexibility our engineers consistently tell us they value.

Job Duties and Responsibilities:

Collaborate for success

• Deliver Databricks ETL projects end to end, from requirements through to pipelines running in production.
• Translate business goals into data solutions and help stakeholders make the right choices about data.
• Contribute to technical decisions, take a significant share of the implementation, and monitor the pipelines you own once they are live.
• Build the One Anova data stream
• Work with real-time telemetry from industrial IoT sensors deployed across the globe.
• Build the BI aggregations that bring data from across platforms together into consistent, reusable data assets.
• Your pipelines are the backbone for our internal natural-language digital assets that lets any employee query Anova's data without writing SQL. The reliability, freshness and clarity of what you publish directly determines whether that experience can be trusted.
• Publish and maintain data assets as master data for the organization.
• Use agentic coding tools — Claude Code, Copilot, Cursor and similar — as a normal part of daily delivery.
• Hold AI-generated code to the same bar as any other code. You are accountable for what you ship.
• Keep repositories, tests and documentation structured so both people and agents can work in them effectively.
• Contribute to and continuously adapt best practices and Ways of Working around data engineering, testing and pipeline operations.
• Maintain clear data lineage and definitions for the assets you own — as AI agents increasingly query this data directly, untraceable or ambiguous data becomes a governance risk, not just a data-quality one.
• Treat data quality as a feature: tests, expectations and monitoring, so problems surface before stakeholders find them.

Minimum Requirements

• Bachelor's degree in Computer Science, Data Engineering, Data Science, or a related quantitative field or equivalent combination of education and experience
• 5+ years of experience in data engineering or a closely related role, with hands‑on production experience in Databricks (6–8 years preferred).
• Significant experience building data workloads in Databricks, with a very good understanding of PySpark and Delta Lake.
• Strong SQL — window functions, complex joins and query tuning are everyday tools for you.
• Experience with streaming or incremental ingestion (Structured Streaming, Auto Loader, or equivalent) and the patterns that keep it correct: idempotency, checkpointing and schema evolution.
• Data modelling for BI and analytics.
• Good understanding of testing and CI/CD for Databricks workflows, alongside the software engineering and DevOps basics — git, code review, linters, unit tests and CI/CD pipelines are things you use daily.
• Data quality practice: testing data as well as code, using pipeline expectations, dbt tests or similar.
• Comfortable using agentic coding tools, with a clear view of where they help and where they need supervision.
• Proficient in written and spoken English.

Preferred Qualifications

• Databricks platform depth beyond the basics: Lakeflow pipelines (formerly Delta Live Tables), Lakeflow Jobs, Unity Catalog for governance and lineage, and infrastructure as code with Declarative Automation Bundles or Terraform.
• Performance and cost optimization on Databricks: cluster sizing, Photon, liquid clustering, and partitioning.
• The wider Azure data ecosystem: Event Hubs or Data Factory.
• Master data management or data governance practice: clear ownership, stewardship and agreed definitions for shared data assets.
• Domain experience in industrial, energy or IoT settings.