Data Engineer: Scalable Ingestion & AI Search Platform

Solveintelligence; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Full visa sponsorship

Private medical insurance

Competitive salary

Job summary

Solve Intelligence is hiring a Data Engineer to build ingestion and search systems behind its AI products in London. You will handle data from sources like patent literature, case law, standards, and scientific databases, enabling scalable search and retrieval across diverse media.

You'll own end-to-end data pipelines, design schemas, and optimize performance while collaborating with AI researchers and product engineers to shape the platform's foundation.

Qualifications

• Strong Python and SQL, with experience designing and operating production databases.
• Experience building and operating production data pipelines over large, messy datasets.
• Expertise with running search systems over large document collections.
• End-to-end ownership from raw data to user-facing functionality.
• A good understanding of schema design, indexing and query optimisation.
• A track record of diagnosing and fixing performance bottlenecks in live systems through profiling and measurement.

Responsibilities

• Large-scale ingestion. Build and operate high-throughput, resumable pipelines for large datasets, with efficient incremental updates, monitoring and recovery from failures.
• Document processing and data quality. Extract useful content from complex documents and other formats. Handle malformed records and changing schemas, and validate outputs while preserving structure and metadata.
• Search and serving. Build keyword, vector and structured search, and design schemas, indexes and partitioning for fast queries over tens to hundreds of millions of records.
• Connecting information across sources. Link patents, scientific records and supporting documents, preserve dates and versions, and make results traceable to their original sources.
• Performance engineering. Profile parsing, ingestion, database builds and queries throughout development, testing against representative datasets at realistic scale. Diagnose CPU, memory and storage I/O bottlenecks, and tune jobs and infrastructure for throughput, latency and cost.

Skills

Python

SQL

Data pipelines

Production databases

Search systems

Schema design

Performance profiling

Tools

PostgreSQL/pgvector

OpenSearch/Elasticsearch

Spark/Delta Lake

AWS

NoSQL databases

Rust/C++

Job description

Solve Intelligence is hiring a Data Engineer to build ingestion and search systems behind its AI products in London. You will handle data from sources like patent literature, case law, standards, and scientific databases, enabling scalable search and retrieval across diverse media.

You'll own end-to-end data pipelines, design schemas, and optimize performance while collaborating with AI researchers and product engineers to shape the platform's foundation.