Web Scraping Engineer (Python)

Adthena; Greater London; Remote; GBP 60,000 - 80,000; Full time

Job summary

A leading search intelligence platform in the United Kingdom is seeking a Web Scrape Engineer to design and manage robust web scraping systems. The ideal candidate will have extensive experience in Python, anti-bot strategies, and cloud environments. This position offers a remote-first approach and ample opportunities for career growth, alongside a competitive compensation package and benefits like stock options and extensive health support.

Benefits

Stock Options

Trust-Based Vacation

Private Medical Insurance

Home Office Stipend

Birthday Day Off

Volunteer Days

Qualifications

• At least 5 years of experience building production web scraping systems.
• Deep understanding of Python and relevant frameworks.
• Experience with deploying workloads in cloud environments.

Responsibilities

• Design and maintain scalable web scraping systems.
• Innovate anti-bot strategies.
• Integrate outputs into data pipelines.
• Ensure the resilience and monitoring of systems.

Skills

Python

Web scraping

Anti-bot strategies

Docker

AWS

Distributed systems

Education

Bachelor's degree in Computer Science or related field

Tools

Scrapy

Selenium

Kafka

Job description

Our Mission

Join us in shaping the future of paid search intelligence.

Adthena is an award-winning search intelligence platform powered by patented AI technology and a team of dedicated experts. We help brands marketers and agencies dominate their competitive landscapes with unparalleled insights into the paid search market. Our Whole Market View Smart Monitor and Local View solutions give clients the clarity to optimize spend increase ROI and stay ahead of the competition.

Trusted by global brands like Citibank LOréal and Volvo and backed by Updata Partners weve been redefining search intelligence since 2012. Ready to make an impact in an industry-leading company Lets do it together.

In 2024 Adthena continued its winning streak earning 9 major search industry awards. Highlights include Best AI Search Software Solution at the Global Search Awards Best Use of AI for Data at the US Search Awards and Best PPC Management Software Suite at the UK Search Awards.

Why Join Us

We offer the best of both worlds : the maturity and stability of a well-established company combined with the energy agility and impact of a scaling tech organisation.

If youve been hesitant about joining a typical tech company due to concerns about chaos fragility or volatility Adthena is your answer.

Were smaller than large-scale enterprises offering more autonomy less red tape and accelerated growth.

Heres what we offer :

• Startup Engineering culture
• Strong Culture : Our quarterly surveys show a 93% average approval rating for company culture with an overall eNPS of 27.
• Award-Winning Product : Best AI Search Software Solution Global Search Award 2024
• Stock Options : Our employees do incredible work so as we grow everyone shares the rewards.
• Trust-Based Vacation : Take as much time off as you need when you need it.
• Remote-First : 52% of Adthenians work fully remote with others choosing Hybrid Work at our 3 Geohubs (London Austin Sydney).
• Flexible Work : Work how and where you do your best with full autonomy over your day.
• Career Growth : Bespoke training and career development via Sherpa plans to guide your growth.
• Monthly Hackdays & Training Days
• Private Medical Insurance : Fully covered health care.
• Mental Health Support : Employee Assistance Program offering 24 / 7 confidential counselling with access to healthier living services such as HeadSpace.
• Family Care Package : Up to 6 months fully paid maternity leave and 2 months paternity leave.
• Pension : Competitive UK pension scheme available from day one.
• Home Office Stipend : $200 for your ideal remote setup.
• Swag Welcome Gift : $50 credit to grab some merch.
• Birthday Day Off : Celebrate your special day.
• Regular Socials : Summer & Christmas parties annual and quarterly offsites monthly meet-ups.
• Volunteer Days : Make a difference with paid volunteer time each quarter.

About the Role

Reporting to : Vasanth John VP of Engineering

Location : United Kingdom

Start Date : ASAP

About your Team

Adthena is a market leader in developing complex ETL and machine learning solutions. With published authors and award winning data scientists we are a friendly passionate group of engineers making a career out of building great software for our customers.

Adthenas scraping team is at the heart of how we capture the web and AI ecosystem. Every day our systems pull in massive volumes of search and advertising data alongside emerging sources like Googles AI Overviews Perplexity and ChatGPT at a scale of hundreds of millions of data points. These scrapers run around the clock orchestrating thousands of processes working across distributed systems and feeding the insights that make our products indispensable.

We are a team that values curiosity and collaboration. Knowledge is openly shared new approaches are encouraged and we support one another in solving the kinds of problems that few companies get to tackle. Its a place where engineers grow by taking ownership experimenting with new tools and building systems that need to withstand constant change.

What youll do at Adthena

As a Web Scrape Engineer youll be the go-to person for web and LLM scraping at Adthena. Youll design and operate resilient pipelines that can outsmart anti-bot defenses adapt to fast-moving platforms and deliver clean data into our core systems. Youll work side by side with Data Engineering and DevOps ensuring scrapers are reliable observable and cost-efficient in the cloud. This role is perfect for someone who thrives on independence loves solving difficult problems at scale and wants their work to have immediate visible impact for global customers.

Responsibilities

Lead design build deploy and maintain scalable resilient web / AI / LLM scraping systems ensuring optimal performance and productivity.

Innovate anti-bot / evasion strategies : proxies headless browsers fingerprinting CAPTCHAs.

Integrate scrape outputs into Adthenas pipeline : cleaning deduplication schema delivery to our warehouse.

Ensure resilience : monitoring alerting retries failure modes fallback strategies.

Work with our cloud infra (containers orchestration) collaborating with DevOps for cost reliability deployment.

Proactively adapt to upstream changes (site changes new anti-bot defenses).

Maintain strong documentation best practices code hygiene; communicate clearly across time zones and teams.

Who we look for

Research shows men apply if they meet 60% of the criteria while women and underrepresented groups apply only if they meet all requirements. If you have the skills but dont meet every requirement reach out wed love to explore how you could be a great fit!

Preferred Experience

Bachelor degree in Computer Science similar technical field of study or equivalent practical experience.

Minimum 5 years of commercial experience building and running production web scraping systems that operate at scale.

Deep Python skills; experience with frameworks such as Scrapy Playwright or Selenium.

Practical anti-bot experience : proxies CAPTCHAs request fingerprinting JavaScript rendering and related evasion strategies.

Experience in distributed systems and pipelines (Spark Airflow) with integration into data warehouses like Redshift and PostgreSQL.

Commercial experience scraping LLM / AI sources such as Perplexity ChatGPT Claude or Gemini.

Experience deploying and managing scraping workloads in cloud environments (AWS / GCP) using Docker and Kubernetes.

Proven ability to build observable and resilient systems with monitoring stacks such as Grafana Prometheus and ELK.

Strong debugging performance optimization reliability focus.

Independent self-starter with excellent communication who owns outcomes end-to-end with minimal oversight.

Bonus Points If You Have :

Awareness of legal ethical and compliance considerations in scraping (e.g. GDPR terms of service).

Familiarity with messaging / queuing systems such as Kafka or RabbitMQ.

Knowledge of infrastructure as code (Terraform Ansible) and CI / CD pipelines (Jenkins GitHub Actions TeamCity).

Experience in cloud cost optimisation and scaling strategies for large-scale scraping.

Our Core Values

All Adthenians are expected to align closely with our Core Values.

What it means...

• Get Shit Done : Were big on taking action owning our challenges and finding solutions.
• Freedom with Responsibility : We take charge of results to recognition. We know that our hustle not only moves the needle for the company but also sets us up for personal success.
• Deliver Remarkable Quality : We care about exceeding expectations in the work we do and aim to always do our best work as individuals and as a team.
• Smart & Always Improving : We strive daily to expand our knowledge and become better at what we do.

Meet your teammates

• Paul Felby
  Title : Chief Product & Tech Officer
  LinkedIn : John
• VP of Engineering
  LinkedIn : Nikolov
• Lead Technical Architect
  LinkedIn : Key Skills
  APIs,Docker,Jenkins,REST,Python,AWS,NoSQL,MySQL,JavaScript,Postgresql,Django,GIT

Employment Type : Full-Time

Experience : years

Vacancy : 1