Software Engineer: Data Pipelines and ML (Mid-career / Senior)

Theia Insights; Cambridge; Hybrid; GBP 90,000 - 140,000; Full time

Benefits offered by this job

EMI share options

25 days holiday + bank holidays

Private health insurance

Pension

Job summary

Theia Insights in the United Kingdom is seeking a data/ML engineer to build and operate production pipelines for global equity classification and knowledge graph work. You’ll run large-scale NLP/LLM inference, ingest market data, and shape schemas and data products used by downstream consumers.

You’ll collaborate with economists and engineers, deploy in a cloud environment, and apply embeddings, clustering and semantic similarity to production datasets, with a focus on reliability and cost

Qualifications

• Strong production Python experience.
• Pandas and Parquet/Arrow for datasets.
• Orchestrated batch pipelines (Dagster or Airflow) and S3-based data flows.
• Applied ML in production—embeddings and semantic similarity.
• AWS fluency and CI/CD discipline.

Responsibilities

• Build and maintain pipelines that classify global public equities across a five-level taxonomy.
• Run large-scale NLP and LLM inference over company documents with batch execution.
• Ingest market data and publish datasets to external distributors.
• Own schema and contract evolution for datasets with downstream consumers.
• Collaborate with economists and engineers to turn modelling decisions into production data.

Skills

Python

Pandas

Parquet/Arrow

DuckDB

Snowflake

Dagster

Airflow

AWS

CI/CD

ML in production

Embeddings

Tools

Snowflake

DuckDB

Job description

About the role

Theia Insights builds foundational financial intelligence products, including industry classification, knowledge graphs and factor risk models, for institutional investors. We serve some of the largest asset managers, hedge funds, index providers and sell-side banks.

What you’ll do

• Build and maintain pipelines that classify global public equities across our five-level taxonomy, sector, industry, sub-industry, major theme and micro theme, by extracting information from filings and web content, and assigning thematic exposures based on this information.
• Run large-scale NLP and LLM inference (entity extraction, classification, knowledge graph construction) over company documents, with cost- and throughput-aware batch execution.
• Ingest market data and publish datasets to external distributors.
• Own schema and contract evolution for datasets with real downstream consumers.
• Work with economists and engineers to turn modelling decisions into reliable production data.

Essential

• Strong production Python.
• Datasets in pandas and Parquet/Arrow, plus an analytical engine, e.g. DuckDB, or a warehouse such as Snowflake.
• Orchestrated batch pipelines you’ve operated, not just written: Dagster or Airflow, S3-based data flows, and a habit of testing outputs for correctness rather than only for exceptions.
• Applied ML in production: embeddings and semantic similarity, clustering, or operationalising models (not necessarily training from scratch).
• AWS fluency and CI/CD discipline.

Nice to have

• Practical LLM engineering: prompting, batch inference, and cost and throughput trade-offs across providers.
• SageMaker, Bedrock, or comparable managed ML tooling.
• Financial and equities domain knowledge: classification taxonomies, factor models, index construction. Valuable but learnable.
• Infrastructure as code (AWS CDK or Terraform) and Docker.

We care more about what you’ve owned than years on a CV. If you’ve built a pipeline that runs on a schedule against real volume, and you were the person who got paged when it broke, you’re in scope. More senior candidates will typically have made the cost and throughput trade-offs, and set the standard for how a team tests data quality.

Experience we’re looking for

• Competitive salary plus EMI share options
• 25 days holiday + bank holidays
• Private health insurance, pension
• Hybrid working in UK from London or Cambridge