Senior Web-Scale Crawling Engineer

United States Digital Space LLC; Greater London; On-site; GBP 70,000 - 90,000; Full time

Benefits offered by this job

Competitive compensation

Career growth and learning opportunities

Flexibility and ownership

Collaborative and innovative culture

Opportunity to work on impactful AI projects

Job summary

United States Digital Space LLC is seeking a Senior Software Engineer based in Greater London to develop system for web-scale crawling of content for AI-powered solutions. Candidates should have extensive experience in backend systems and proficiency in Go or C++. The role focuses on building reliable infrastructure capable of processing vast amounts of data effectively.

Alongside competitive compensation, the position offers opportunities for career growth and involvement in impactful AI projects within a collaborative environment.

Qualifications

• 5+ years of experience building backend or distributed systems.
• Strong expertise in Go or C++.
• Understanding of web protocols and content extraction.

Responsibilities

• Design and implement web-scale crawling systems.
• Build ingestion workflows for data sources.
• Ensure reliable operation under high-throughput conditions.

Skills

Go

C++

Distributed systems

Scalability knowledge

Web protocols (HTTP, DNS, TLS)

Tools

Kafka

Spark

Flink

Job description

United States Digital Space LLC is seeking a Senior Software Engineer based in Greater London to develop system for web-scale crawling of content for AI-powered solutions. Candidates should have extensive experience in backend systems and proficiency in Go or C++. The role focuses on building reliable infrastructure capable of processing vast amounts of data effectively.

Alongside competitive compensation, the position offers opportunities for career growth and involvement in impactful AI projects within a collaborative environment.