Python Web Scrapping Engineer

Tenth Revolution Group; Remote; GBP 70,000 - 90,000; Full time

Job summary

A leading tech company in the UK is seeking a Senior Python Scraping Engineer to design and maintain large-scale scraping systems, particularly focused on Google. The ideal candidate has strong Python experience, a deep understanding of scraping protected platforms, and excellent skills in handling complex scraping challenges. This role offers the opportunity to work with advanced AI tools while ensuring data accuracy and usability. The position is perfect for those passionate about innovative coding practices in an agile environment.

Qualifications

• Strong professional experience with Python and production-grade scraping.
• Experience scraping Google or similarly protected platforms.
• Solid understanding of HTTP, TLS, cookies, and networking behavior.
• Comfortable handling rate limiting and bot detection systems.
• Experience with cloud environments and running scraping systems at scale.

Responsibilities

• Design, build, and maintain large-scale Python scraping systems.
• Develop scrapers for dynamic pages using browser automation.
• Engineer robust extraction pipelines prioritizing data accuracy.
• Monitor scrape health and debug production issues promptly.
• Collaborate with teams to ensure usability of scraped data.

Skills

Python

Scraping techniques

HTTP fundamentals

Browser automation

Data extraction methods

Asynchronous programming

Debugging skills

Tools

Playwright

Selenium

Puppeteer

Job description

Overview

We are looking for a Senior Python Scraping Engineer to design, build, and operate large-scale, resilient web scraping systems, with a strong focus on Google and Google-like search environments. This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale. You will work on complex scraping problems where naïve approaches do not work - dynamic rendering, sophisticated anti-bot defences, frequent markup changes, and high-volume data collection. We are leaders in adopting AI‑assisted and agentic coding workflows. This role is ideal for someone who believes in leveraging AI tools to accelerate development and wants to deepen their skills in this area.

What You’ll Do

• Design, build, and maintain large‑scale Python scraping systems that operate reliably against Google and other highly defended websites
• Develop scrapers for dynamic, JavaScript‑heavy pages using browser automation and hybrid scraping approaches
• Continuously adapt to changes in page structure, request flows, and anti‑automation mechanisms
• Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability
• Implement and optimise proxy, fingerprinting, and request‑routing strategies to maximise stability and throughput
• Monitor scrape health, detect failures early, and debug production issues quickly and methodically
• Optimise performance, cost, and latency across scraping infrastructure
• Collaborate closely with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted
• Document scraping logic, architectural decisions, and operational runbooks for long‑term maintainability

Required Skills and Experience

• Strong professional experience with Python, including production‑grade scraping systems
• Deep, hands‑on experience scraping Google or similarly protected platforms
• Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour
• Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalents
• Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques
• Proven experience handling rate limiting, CAPTCHAs, IP rotation, and bot detection systems
• Comfortable designing and operating asynchronous and concurrent scraping architectures
• Experience running scraping systems at scale in cloud environments
• Excellent debugging skills and the ability to reason about complex, non‑obvious failure modes
• Hands‑on experience with headless browser fingerprinting and anti‑bot evasion techniques

To apply for this role please submit your CV or contact Dillon Blackburn on (phone number removed) or at (url removed).