Senior Web Crawling Engineer

Hexanix One; Ahmedabad District; Hybrid; INR 4,000,000 - 6,000,000; Full time

Benefits offered by this job

Hybrid work model

Professional development

Team building events

PTO & holidays

Job summary

Hexanix One is hiring a Senior Web Crawling Engineer to build and maintain the crawling infrastructure powering our data extraction and analytics platforms. You will design robust Scrapy-based crawlers, deploy distributed solutions with Redis and Kafka, and integrate headless browsers for JS-heavy sites.

We offer a hybrid work setup with remote flexibility and market-aligned compensation; you will optimize crawler performance and bypass anti-scraping measures at scale in a fast-paced environment.

Qualifications

• 5+ years of Python development experience.
• Hands-on Scrapy experience with core framework, pipelines, middleware, extensions.
• Deep understanding of Scrapy internals (Twisted, asynchronous networking).
• Experience handling dynamic websites and API-based scraping.
• Strong knowledge of XPath, CSS selectors, JSONPath.

Responsibilities

• Architect, design, and implement scalable web crawlers using Scrapy.
• Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers.
• Develop distributed crawling solutions using Redis/Kafka-based queues.
• Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management).
• Implement fingerprinting, proxy rotation and user-agent rotation.
• Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy sites.

Skills

Python programming

Scrapy experience

Asynchronous networking

Web scraping

XPath/JSONPath

HTTP/HTTPS fundamentals

Tools

Scrapy

Twisted

Playwright

Selenium

Redis

Kafka

Job description

Back to All Jobs

Senior Web Crawling Engineer

Engineering Ahmedabad / Remote Full-time as per the market standards

About The Role

We're looking for a Senior Web Crawling Engineer to help build and maintain the web crawling infrastructure that powers our data extraction and analytics platforms. You'll work on developing robust, scalable crawlers that extract valuable data from complex websites and APIs.

Key Responsibilities

• Architect, design, and implement scalable web crawlers using Scrapy framework as the primary technology
• Build reusable Scrapy spiders, middlewares, pipelines, extensions, and custom schedulers
• Develop distributed crawling solutions using Scrapy Cluster / Redis / Kafka-based queue systems
• Optimize Scrapy performance (auto-throttling, concurrency tuning, memory management)
• Implement advanced request fingerprinting, retry logic, proxy rotation, and user-agent rotation
• Implement strategies to bypass anti-scraping mechanisms (rate limiting, CAPTCHA handling, fingerprinting)
• Work with headless browsers (Playwright/Selenium) integrated with Scrapy for JS-heavy websites

Required Qualifications

• 5+ years of strong Python development experience
• Extensive hands-on experience with Scrapy (core framework, pipelines, middleware, extensions)
• Deep understanding of Scrapy internals (Twisted, asynchronous networking)
• Strong experience with XPath, CSS selectors, JSONPath
• Solid understanding of HTTP/HTTPS protocols, headers, cookies, sessions
• Experience handling dynamic websites and API-based scraping

Nice to Have

• Experience integrating Scrapy with Playwright or Selenium
• Knowledge of bot detection techniques and mitigation
• Experience building REST APIs for crawl orchestration
• Familiarity with cloud deployment and DevOps practices

What We Offer

Competitive compensation package

Hybrid work model with flexible hours

Professional development opportunities

Team building events and activities

Generous PTO and holiday schedule