Scraper Developer (Swiss Based MNC Co.)

Recruise India Consulting; Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Recruise India Consulting is seeking a Scraper Developer to build and maintain web scraping solutions for its competitive intelligence platform. You will analyze websites, identify data sources, and develop robust extraction pipelines for product and pricing data at scale.

The role involves creating and maintaining scrapers with Scrapy, BeautifulSoup, lxml, and modern browser automation tools, plus writing unit tests with pytest to ensure data accuracy.

Qualifications

• Proficient in Python with hands-on scraping experience.
• Strong knowledge of HTML, CSS, and data extraction.
• Experience building scalable scrapers for ecommerce or competitors.
• Familiar with XPath, CSS selectors and JSON data parsing.
• Experience with Playwright and Selenium for dynamic pages.
• Able to write clean, tested Python code and unit tests.

Responsibilities

• Develop and maintain web scrapers for competitor websites.
• Extract structured data: product details, pricing, availability, and URLs.
• Analyze sites to identify HTML, API, AJAX, and JS-rendered data sources.
• Build scrapers using Scrapy, BeautifulSoup, lxml, requests, Playwright, and Selenium.
• Handle pagination, dynamic content, API integrations, and data normalization.
• Write maintainable Python code and unit tests with pytest.
• Troubleshoot scraper issues from website structure changes.
• Collaborate with QA and engineering to resolve data quality issues.

Skills

Python

XPath

CSS Selectors

JSON parsing

Git

Debugging techniques

Chrome DevTools

pytest

Tools

Scrapy

BeautifulSoup4

lxml

requests

Playwright

Selenium

Job description

Role & responsibilities

We are looking for a passionate Scraper Developer to build and maintain web scraping solutions that support our competitive intelligence platform. The ideal candidate enjoys analyzing websites, identifying data sources, and developing robust extraction pipelines to collect product and pricing information at scale.

Key Responsibilities

• Develop and maintain web scrapers for competitor websites.
• Extract structured data including product details, pricing, availability, categories, and URLs.
• Analyze websites using Chrome DevTools to identify HTML, API, AJAX, and JavaScript-rendered data sources.
• Build scraping solutions using Scrapy, BeautifulSoup, lxml, requests, Playwright, and Selenium.
• Handle pagination, dynamic content, API integrations, and data normalization.
• Write clean, maintainable, and well-documented Python code.
• Create and execute unit tests using pytest to ensure data accuracy and reliability.
• Troubleshoot and fix scraper issues caused by website structure changes.
• Collaborate with QA and engineering teams to resolve data quality issues.

Required Skills

• Strong experience with Python
• Hands-on experience with Scrapy, BeautifulSoup4, lxml, and requests
• Good knowledge of XPath, CSS Selectors, and JSON parsing
• Experience with Playwright and/or Selenium
• Familiarity with Chrome DevTools (Network & Elements tabs)
• Knowledge of pytest, Git, and debugging techniques
• Understanding of web technologies including HTML, CSS, JavaScript, and APIs

Preferred Skills

• Experience with anti-bot techniques, proxy rotation, and cookie handling
• Knowledge of CAPTCHA-solving integrations
• Experience scraping e-commerce, retail, travel, or comparison-shopping websites
• Basic understanding of SQL and Docker
• Active GitHub profile showcasing scraping projects