Senior Software Engineer, Data Engineering

United States Digital Space LLC; Greater London; Hybrid; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Competitive salary and equity

Bonuses and benefits

Health care and retirement plans

Employee giving match

Mobile phone stipend

R&R days

Wellness reimbursement

Generous vacation policy

Parental leave

Team offsites and events

Job summary

Caspian is building a centralized data platform to power analytics and business operations. As a Senior Software Engineer, you will own end-to-end data pipelines, model design, and observability across multiple domains while mentoring peers and shaping platform infrastructure.

You'll work with Databricks, Delta Live Tables, and Unity Catalog, leveraging Python and SQL to deliver scalable, cost-efficient data solutions with strong governance and data contracts.

Qualifications

• 5–8 years of hands-on data engineering experience building production pipelines and data models.
• Proficiency with Databricks or similar platforms, Delta Live Tables, Unity Catalog, Delta Lake, PySpark or Spark SQL.
• Strong SQL skills including transformations, incremental materialization, and data quality validation.
• Solid AWS experience for building and operating data systems at scale.
• Proficiency in Python for data engineering tasks.
• Experience with CI/CD tooling (GitLab) and infrastructure-as-code (Terraform).
• Exposure to AI tooling in data context (agents, code generation, self-serve analytics).
• Ability to translate ambiguous requirements into robust data models and pipelines with minimal oversight.

Responsibilities

• Design, build, and operate production ETL pipelines on Databricks using Delta Live Tables and Spark.
• Own data modeling in governed, reusable table structures serving as truth sources for metrics.
• Drive technical design for cross-team projects with clear architecture and rollout plans.
• Implement pipeline observability: failure detection, alerts, retries, operational hygiene.
• Integrate AI tooling into data workflows for automated transformations and data access.
• Mentor engineers, review code and designs, help grow ownership.
• Contribute to platform infrastructure via Terraform and manage Unity Catalog data contracts and access.

Skills

Databricks

Delta Live Tables

Unity Catalog

Delta Lake

PySpark

Spark SQL

Python

AWS

CI/CD

Terraform

Tools

GitLab

Terraform

Fivetran

Job description

At the company, we’re building a world where value moves like information does today. It’s big, it’s bold, and we’re already doing it. Through our crypto solutions for financial institutions, businesses, governments and developers, we are improving the global financial system and creating greater economic fairness and opportunity for more people, in more places around the world. And we get to do the best work of our career and grow our skills surrounded by colleagues who have our backs.

If you’re ready to see your impact and unlock incredible career growth opportunities, join us, and build real world value.

Role Overview

At the company, we're building a world where value moves like information does today. It's big, it's bold, and we're already doing it. Through our crypto solutions for financial institutions, businesses, governments and developers, we are improving the global financial system and creating greater economic fairness and opportunity for more people, in more places around the world. And we get to do the best work of our career and grow our skills surrounded by colleagues who have our backs.

If you're ready to see your impact and unlock incredible career growth opportunities, join us, and build real world value.

As a Senior Software Engineer on the Data Engineering team, you are a core builder of the Caspian Data Platform — the company's centralized lakehouse that powers analytics, financial reporting, product intelligence, and data-driven operations across every business unit. You own the design and delivery of production-grade pipelines end to end. You raise the engineering bar for the team through code reviews, technical design, and mentorship. You operate with high autonomy on complex, cross-functional data initiatives and bring sound judgment to tradeoffs around performance, cost, reliability, and maintainability.

What you'll do

• Design, build, and operate production ETL pipelines on Databricks using Delta Live Tables and Spark, applying medallion architecture patterns across Finance, Payments, Product, and GTM domains.

• Own data modeling in different layers — defining governed, reusable table structures that serve as reliable sources of truth for business metrics and downstream consumption.

• Drive technical design for projects spanning multiple teams, writing clear design documents that articulate architecture, tradeoffs, and rollout plans.

• Build pipeline observability into your work — failure detection, alerting, retry logic, and the operational hygiene that keeps oncall burden low.

• Integrate AI tooling into data workflows: agentic systems for pipeline triage, automated code generation for transformations, and natural language interfaces for data access.

• Mentor engineers, reviewing their code and designs and helping them grow into more complex ownership.

• Contribute to platform infrastructure through Terraform, and work with Unity Catalog to establish data contracts, access controls, and lineage.

What you'll bring

• 5–8 years of hands‑on data engineering experience building and maintaining production pipelines and data models.

• Deep proficiency with Databricks or similar platforms — Delta Live Tables, Unity Catalog, Delta Lake, and PySpark or Spark SQL as primary tools.

• Strong SQL skills: complex transformations, incremental materialization, schema evolution, and data quality validation.

• Solid AWS experience relevant to building and operating data systems at scale.

• Proficiency in Python for data engineering — ingestion, transformation, testing, and utility tooling.

• Experience with CI/CD tooling such as GitLab and infrastructure‑as‑code with Terraform.

• Exposure to AI tooling in a data context — whether building agents, using LLMs for code generation, or enabling self‑serve analytics workflows.

• The ability to take ambiguous requirements, define the right data model and pipeline design, and deliver with minimal oversight.

Nice to have

• Familiarity with financial, payments, or blockchain data.

• Experience with Fivetran or similar ingestion tools and their operational patterns.

WHO WE ARE:

Do Your Best Work

• The opportunity to build in a fast-paced start-up environment with experienced industry leaders

• A learning environment where you can dive deep into the latest technologies and make an impact. A professional development budget to support other modes of learning.

• Thrive in an environment where no matter what race, ethnicity, gender, origin, or culture they identify with, every employee is a respected, valued, and empowered part of the team.

• In‑office collaboration for moments that matter is important to our culture, and we give managers and teams the flexibility to decide which 10+ days a month they come in.

• Bi‑weekly all-company meeting - business updates and ask me anything style discussion with our Leadership Team

• We come together for moments that matter which include team offsites, team bonding activities, happy hours and more!

Take Control of Your Finances

• Competitive salary, bonuses, and equity

• Competitive benefits that cover physical and mental healthcare, retirement, family forming, and family support

• Employee giving match

• Mobile phone stipend

Take Care of Yourself

• R&R days so you can rest and recharge

• Generous wellness reimbursement and weekly onsite & virtual programming

• Generous vacation policy - work with your manager to take time off when you need it

• Industry‑leading parental leave policies. Family planning benefits.

• Catered lunches, fully‑stocked kitchens with premium snacks/beverages, and plenty of fun events

*Benefits listed above are for full‑time employees.*

*the company is an Equal Opportunity Employer. We’re committed to building a diverse and inclusive team. We do not discriminate against qualified employees or applicants because of race, color, religion, gender identity, sex, sexual identity, pregnancy, national origin, ancestry, citizenship, age, marital status, physical disability, mental disability, medical condition, military status, or any other characteristic protected by local law or ordinance.*

*Please find our UK/EU Applicant Privacy Notice and our California Applicant Privacy Notice for reference.*

Find more English Speaking Jobs in United Kingdom on Arbeitnow