Web Data Extraction Engineer

Crossing Hurdles; United Kingdom; On-site; GBP 45,000 - 65,000; Full time

Job summary

Crossing Hurdles is seeking a skilled web scraping engineer to design, develop, and maintain robust scraping pipelines that extract data from diverse online sources.

You will handle static and dynamic content, clean and structure large datasets, monitor pipelines for quality, troubleshoot issues, and document processes to enable scalable reuse. Collaborate with teammates to define data requirements and ensure deliverables align with project goals while adhering to web security best practices.

Qualifications

• Have proven experience in building and maintaining web scrapers using tools such as Python, Scrapy, Selenium, or similar technologies.
• Have expertise in handling APIs, RESTful interfaces, and parsing JSON/XML data.
• Demonstrate proficiency with data cleaning, manipulation, and storage using relational or NoSQL databases.

Responsibilities

• Design, develop, and maintain robust web scraping scripts and applications to extract data from various online sources.
• Perform data extraction from multiple websites, effectively handling both static and dynamic content.
• Clean, validate, and structure large volumes of scraped data to ensure accuracy for further analysis.
• Monitor scraping pipelines for data quality, troubleshoot scraping failures, and implement solutions to maintain effectiveness.
• Collaborate with team members to define data requirements and deliverables, ensuring alignment with project goals.
• Document scraping processes and maintain code for scalability and reusability in future projects.

Skills

Web scraping

Data extraction

Data cleaning

Tools

Python

Scrapy

Selenium

Job description

• Design, develop, and maintain robust web scraping scripts and applications to extract data from various online sources.
• Perform data extraction from multiple websites, effectively handling both static and dynamic content.
• Clean, validate, and structure large volumes of scraped data to ensure accuracy for further analysis.
• Monitor scraping pipelines for data quality, troubleshoot scraping failures, and implement solutions to maintain effectiveness.
• Collaborate with team members to define data requirements and deliverables, ensuring alignment with project goals.
• Document scraping processes and maintain code for scalability and reusability in future projects.

Requirements

• Have proven experience in building and maintaining web scrapers using tools such as Python, Scrapy, Selenium, or similar technologies.
• Have expertise in handling APIs, RESTful interfaces, and parsing JSON/XML data.
• Demonstrate proficiency with data cleaning, manipulation, and storage using relational or NoSQL databases.
• Have knowledge of web security concepts and best practices for ethical scraping.

Application Process

• Easy Apply on LinkedIn
• Participate in resume evaluation & interview stage