Senior Data Engineer at Causeway Technologies

Causeway Technologies; Lisboa; Presencial; EUR 65 000 - 85 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Hybrid working environments

Focus on work-life balance

Opportunities for career growth

Resumo da oferta

Causeway Technologies is seeking a Senior Data Engineer in Lisbon (Hybrid). This pivotal role involves building a next-generation data platform on Databricks, handling data unification from numerous sources into a resolvable structure suitable for AI agents. Candidates should possess strong experience with Databricks, Python, and cloud storage solutions.

Key responsibilities include managing the medallion pipeline, ensuring data quality, and providing technical leadership within the team. The position offers opportunities for career growth in an award-winning environment, with a focus on well-being and work-life balance.

Qualificações

• Strong production experience on Databricks, Delta Lake, and Delta Live Tables.
• Deep proficiency in Python and PySpark for large-scale data processing.
• Advanced SQL skills including recursive CTEs and query optimization.
• Experience with data transformation and validation frameworks.

Responsabilidades

• Own ingestion and maintain the medallion pipeline layers.
• Monitor data quality and governance across pipeline stages.
• Lead architectural discussions and mentor junior engineers.
• Collaborate with cross-functional teams on data projects.

Conhecimentos

Databricks

Python

PySpark

SQL

PostgreSQL

Data Quality

Data Governance

Cloud Storage

AI Agents

Descrição da oferta de emprego

Senior Data Engineer – Lisbon (Hybrid)

Purpose: The Senior Data Engineer plays a central role in building Causeway’s next‑generation data platform on Databricks, unifying data from across the Causeway product estate and third‑party sources into a typed, entity‑resolved digital twin that AI agents can reason over. The Senior Data Engineer owns significant slices of the medallion pipeline (bronze to silver to gold), the modelling layer that turns raw Delta tables into typed object and relationship tables, and the serving layer that makes the graph queryable by agents with interactive latency. The role is expected to design for scale, set the bar for engineering quality across the team, and make architectural decisions that are durable.

Key Responsibilities

• Medallion Pipeline & Graph Engineering
  • Design and own ingestion into the bronze layer from a variety of sources, selecting the appropriate pattern per source (Auto Loader, API pull, event‑driven, database replica/CDC, CQRS read models) and agreeing data contracts with Architects and Product teams.
  • Build and maintain the silver and gold medallion layers as Delta Live Tables and PySpark notebooks, handling deduplication, entity resolution (deterministic and probabilistic), canonical ID assignment, and projection into typed object and relationship tables.
  • Generate and maintain the typed relationship graph that transforms disconnected gold tables into a traversable digital twin.
  • Own and evolve the serving layer that makes the graph queryable by AI agents at interactive latency.
  • Contribute to the design and evolution of the broader Causeway data platform architecture, ensuring solutions are scalable, maintainable, and aligned with cloud‑native best practices.
• Data Quality & Governance
  • Own the data quality and testing story end‑to‑end, implementing automated frameworks that validate completeness, accuracy, consistency, and schema conformance across pipeline stages.
  • Deliver governance via Unity Catalog, covering multi‑tenant isolation, row‑level security, lineage, audit logging, and the tenant‑scoped extension schema that supports custom objects from external customer systems.
  • Ensure data assets are catalogued correctly with appropriate metadata, security classifications, and sensitivity labels in line with company data governance standards.
  • Monitor pipeline health and data quality metrics, proactively identifying and resolving issues before they affect downstream consumers or agent behaviour.
• Technical Leadership & Collaboration
  • Mentor engineers across the data team, lead architectural discussions, conduct PR reviews, and challenge poor patterns, including those already in production.
  • Partner with the Platform and Agent teams on the tool layer that exposes the graph to AI agents, ensuring datasets are agent‑legible through rich column comments, typed relationships, embedding templates, and governed tooling.
  • Work with Data Engineers to translate complex analytical requirements into structured, reusable datasets and features.
  • Contribute to the continuous improvement of data engineering practices, tooling, and standards across the team.
  • Take responsibility for personal development, staying current with emerging technologies in data engineering, lakehouse architecture, and AI agent tooling.

Key Skills, Experience and Qualifications

• Strong production experience on Databricks, including Delta Lake, Delta Live Tables, Auto Loader, Unity Catalog, Databricks Asset Bundles, serverless and job compute, and Structured Streaming for near‑real‑time workloads.
• Deep proficiency in Python and PySpark for large‑scale data processing and transformation.
• Advanced SQL, including recursive CTEs for graph traversal, window functions, query planning, and the ability to interpret EXPLAIN output and optimise index usage.
• Strong experience with lakehouse architectures and data modelling for graph workloads – typed entities, edge tables, dimensional vs event‑style modelling, and entity resolution across systems that do not share keys.
• Hands‑on experience with PostgreSQL as a serving layer, including pgvector for semantic search, pg_trgm for fuzzy matching, HNSW vs IVFFlat trade‑offs, index tuning, and managing interactive‑latency queries under concurrency.
• Expertise in data transformation, validation, and contract design.
• Experience building data test and observability frameworks.
• Working knowledge of cloud storage and identity across AWS, Azure, and GCP – including S3, ADLS Gen2, GCS, cross‑cloud copy, IAM roles, and storage credentials.
• Genuine curiosity about AI agents and how they consume data, with an understanding of what makes a dataset agent‑legible.
• Desirable – Experience with probabilistic entity resolution techniques and tools.
• Familiarity with knowledge graph concepts and graph query languages.
• Exposure to AI/ML workflows and feature engineering for machine learning models.

Benefits

• Hybrid working with the option to work from modern, collaborative offices.
• Prioritisation of well‑being and work‑life balance.
• Opportunities for career growth in an award‑winning company.