Senior Software Developer

Intrics Solutions; Dadri, Hyderabad; On-site; INR 600,000 - 900,000; Full time

Job summary

Intrics Solutions is seeking a skilled web scraping engineer to design, develop, and maintain scalable scraping solutions in Python. You will work with Scrapy, Selenium, Playwright, BeautifulSoup, and Requests to extract data from diverse websites, including JS-rendered pages.

You will tackle anti-bot challenges, implement robust data validation, and build REST APIs for data processing. This role demands strong debugging skills and the ability to manage end-to-end crawling projects in a

Qualifications

• Experience with large-scale e-commerce crawling.
• Understanding of website request/response cycles and browser dev tools.
• Ability to independently manage end-to-end scraping projects.
• Strong debugging and troubleshooting skills.
• Experience with cloud environments and distributed crawling.

Responsibilities

• Design, develop, and maintain scalable web scraping solutions using Python.
• Extract data from static, dynamic, and JS-rendered sites.
• Handle anti-bot measures, CAPTCHA challenges, proxies, and session management.
• Develop and integrate REST APIs for data extraction and processing.
• Monitor, schedule, and manage large-scale crawling jobs; ensure data quality.

Skills

Python

Debugging

Analytical thinking

Tools

Scrapy

Selenium

Playwright

BeautifulSoup

Requests

Job description

Role & responsibilities

• Design, develop, and maintain scalable web scraping and crawling solutions using Python.
• Extract data from complex websites, including static, dynamic, and JavaScript-rendered websites.
• Work with tools and frameworks such as Scrapy, Selenium, Playwright, BeautifulSoup, and Requests.
• Handle website anti-bot mechanisms, CAPTCHA challenges, IP blocking, proxies, cookies, and session management.
• Develop and integrate REST APIs for data extraction and processing.
• Analyze website structures, network requests, API endpoints, and JSON responses.
• Optimize scraping scripts for performance, scalability, and reliability.
• Troubleshoot crawling failures, identify root causes, and implement permanent fixes.
• Manage large-scale crawling jobs, scheduling, monitoring, and error handling.
• Ensure data accuracy, completeness, and consistency through appropriate validation mechanisms.
• Collaborate with QA and operations teams for smooth project execution.
• Provide technical guidance and support to junior developers.

Preferred candidate profile

• Experience in large-scale e-commerce website crawling.
• Knowledge of retail websites and product data extraction.
• Knowledge of cloud environments and distributed crawling.
• Understanding of website request/response cycles and browser developer tools.
• Ability to independently handle end-to-end scraping projects.
• Strong debugging and analytical skills.
• Good problem-solving and troubleshooting abilities.
• Ability to estimate development timelines and meet delivery commitments.
• Ownership of assigned projects and production issues.
• Ability to guide junior team members and share technical knowledge.