Senior Web Crawling Engineer

Hexanix One; Ahmedabad District; Hybrid; INR 1,500,000 - 2,100,000; Full time

Benefits offered by this job

Hybrid work model with flexible hours

Professional development opportunities

Generous PTO and holidays

Job summary

Hexanix One is seeking a Senior Web Crawling Engineer to design and maintain scalable crawlers that power data extraction and analytics platforms. You will build reusable Scrapy components, implement distributed queues, and optimize performance for JS-heavy sites with Playwright or Selenium.

The role involves bypassing anti-scraping measures and working with modern tech stacks like Redis, Kafka, and Scrapy Cluster in a hybrid work environment. 5+ years of Python/Scrapy expertise required.

Qualifications

• 5+ years of Python development experience.
• Extensive hands-on Scrapy experience (core framework, pipelines, middleware).
• Deep understanding of Scrapy internals (Twisted, async networking).
• Strong experience with XPath, CSS selectors, JSONPath.
• Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions.

Responsibilities

• Architect, design, and implement scalable web crawlers using Scrapy as the primary technology.
• Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers.
• Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queues.
• Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management).
• Implement advanced request fingerprinting, retry logic, proxy and user-agent rotation.
• Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting).
• Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy sites.

Skills

Python

Scrapy

Twisted

XPath/CSS/JSONPath

HTTP/HTTPS

Dynamic pages & API scraping

Tools

Playwright/Selenium

Redis

Kafka

Scrapy Cluster

Job description

Back to All Jobs

Senior Web Crawling Engineer

Engineering Ahmedabad / Remote Full-time as per the market standards

About The Role

We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs.

Key Responsibilities

• Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology
• Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers
• Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems
• Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management)
• Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation
• Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting)
• Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy websites

Required Qualifications

• 5+ years of strong Python development experience
• Extensive hands‑on experience with Scrapy (core framework, pipelines, middleware, extensions)
• Deep understanding of Scrapy internals (Twisted, asynchronous networking)
• Strong experience with XPath, CSS selectors, JSONPath
• Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions
• Experience handling dynamic websites and API-based scraping

Nice to Have

• Experience integrating Scrapy with Playwright or Selenium
• Knowledge of bot detection techniques and mitigation
• Experience building REST APIs for crawl orchestration
• Familiarity with cloud deployment and DevOps practices

What We Offer

• Competitive compensation package
• Hybrid work model with flexible hours
• Professional development opportunities
• Team building events and activities
• Generous PTO and holiday schedule