Senior Data Engineer: Databricks, Pipelines & AI Tools

United States Digital Space LLC; Greater London; Hybrid; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Competitive salary and equity

Bonuses and benefits

Health care and retirement plans

Employee giving match

Mobile phone stipend

R&R days

Wellness reimbursement

Generous vacation policy

Parental leave

Team offsites and events

Job summary

Caspian is building a centralized data platform to power analytics and business operations. As a Senior Software Engineer, you will own end-to-end data pipelines, model design, and observability across multiple domains while mentoring peers and shaping platform infrastructure.

You'll work with Databricks, Delta Live Tables, and Unity Catalog, leveraging Python and SQL to deliver scalable, cost-efficient data solutions with strong governance and data contracts.

Qualifications

• 5–8 years of hands-on data engineering experience building production pipelines and data models.
• Proficiency with Databricks or similar platforms, Delta Live Tables, Unity Catalog, Delta Lake, PySpark or Spark SQL.
• Strong SQL skills including transformations, incremental materialization, and data quality validation.
• Solid AWS experience for building and operating data systems at scale.
• Proficiency in Python for data engineering tasks.
• Experience with CI/CD tooling (GitLab) and infrastructure-as-code (Terraform).
• Exposure to AI tooling in data context (agents, code generation, self-serve analytics).
• Ability to translate ambiguous requirements into robust data models and pipelines with minimal oversight.

Responsibilities

• Design, build, and operate production ETL pipelines on Databricks using Delta Live Tables and Spark.
• Own data modeling in governed, reusable table structures serving as truth sources for metrics.
• Drive technical design for cross-team projects with clear architecture and rollout plans.
• Implement pipeline observability: failure detection, alerts, retries, operational hygiene.
• Integrate AI tooling into data workflows for automated transformations and data access.
• Mentor engineers, review code and designs, help grow ownership.
• Contribute to platform infrastructure via Terraform and manage Unity Catalog data contracts and access.

Skills

Databricks

Delta Live Tables

Unity Catalog

Delta Lake

PySpark

Spark SQL

Python

AWS

CI/CD

Terraform

Tools

GitLab

Terraform

Fivetran

Job description

Caspian is building a centralized data platform to power analytics and business operations. As a Senior Software Engineer, you will own end-to-end data pipelines, model design, and observability across multiple domains while mentoring peers and shaping platform infrastructure.

You'll work with Databricks, Delta Live Tables, and Unity Catalog, leveraging Python and SQL to deliver scalable, cost-efficient data solutions with strong governance and data contracts.