Databricks Data Engineer: Azure Lakehouse & AI Pipelines

Catena Operations Ltd.; Malta (MT); On-site; USD 90,000 - 140,000; Full time

Job summary

Catena Operations Ltd. is seeking a Data Engineer to join our Data & Analytics team. You will design, build, test and maintain scalable data pipelines and models on our Databricks lakehouse on Azure, governed by Unity Catalog.

You will work in an AI-first environment, contribute to technical design, implement streaming data, ensure data quality, tests and documentation, and collaborate with engineers, analysts and stakeholders.

Qualifications

• Degree in IT, Computer Science, or related field.
• Proven experience building and maintaining scalable ETL/ELT pipelines for data lakes or lakehouses, with batch and real-time data.
• Hands-on Databricks on Azure experience, including Unity Catalog (catalogs, schemas, permissions, lineage).
• Experience with event streaming and real-time ingestion: Spark Structured Streaming and Kafka.
• Strong SQL skills and complex queries.
• Experience with Databricks Workflows and deploying with Databricks Asset Bundles; Azure Data Factory is a plus.
• Proficient in PySpark and Python.
• Solid software engineering practices: Git, PRs, code review, CI/CD, and documentation.
• Hands-on experience with agentic AI tools in daily engineering work.
• Comfortable in a fast-paced, Agile environment.
• Strong analytical and problem-solving skills.
• Excellent written and spoken communication and teamwork.
• Self-motivated with a drive for continuous learning.

Responsibilities

• Design and build data pipelines in Python (PySpark) on Databricks with performance and reliability in mind.
• Ingest diverse data sources into the lakehouse, following bronze/silver/gold layering and ensuring data quality.
• Develop streaming pipelines for real-time ingestion using Kafka and Spark Structured Streaming.
• Maintain data models and the data estate with Unity Catalog governance.
• Write unit and integration tests and data quality checks for code changes.
• Follow engineering standards: modular code, peer review, CI/CD, IaC (Databricks Asset Bundles) and documentation.
• Leverage AI tools to design, test, and troubleshoot data workflows with human review.
• Monitor Databricks on Azure, track pipelines, troubleshoot issues, support daily data operations.
• Collaborate with engineers, analysts, and stakeholders; handle ad hoc data analysis requests.
• Pursue continuous learning and contribute to process improvements.
• Perform other tasks as required by the business.

Skills

ETL pipelines

SQL

Python

PySpark

Testing mindset

CI/CD

Agile environment

Analytical thinking

Communication

Team collaboration

Problem solving

AI tooling

Education

Degree in IT/Computer Science or related field

Tools

Databricks on Azure

Unity Catalog

Kafka

Spark Structured Streaming

Azure Data Factory

Git

GitHub Actions

Azure DevOps

Databricks Workflows

Job description

Catena Operations Ltd. is seeking a Data Engineer to join our Data & Analytics team. You will design, build, test and maintain scalable data pipelines and models on our Databricks lakehouse on Azure, governed by Unity Catalog.

You will work in an AI-first environment, contribute to technical design, implement streaming data, ensure data quality, tests and documentation, and collaborate with engineers, analysts and stakeholders.