Data Engineer Web Scraping Crawling

OSI Digital, Inc.; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

OSI Digital, Inc. in Hyderabad is seeking a Web Crawling Engineer with strong Python expertise to build scalable crawlers and scrapers for data extraction.

The ideal candidate will handle anti-blocking techniques, proxy rotation, and CAPTCHA challenges while collaborating with the data engineering team to integrate crawled data into pipelines. 3-5 years of experience required.

Qualifications

• Strong Python programming skills.
• Experience with Scrapy, Requests, BeautifulSoup, Selenium.
• Experience solving blocking issues and captchas.
• Knowledge of proxy rotation and fingerprinting avoidance.
• Understanding of HTTP, cookies, headers, sessions.
• Experience with SQL/NoSQL databases for crawling data.
• Strong debugging and analytical abilities.

Responsibilities

• Design, develop, and maintain scalable web crawlers for data extraction.
• Use Scrapy, Requests, and other Python libraries to implement crawling solutions.
• Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
• Implement and manage proxy rotation and session strategies.
• Ensure high-quality and structured data extraction, cleaning, and storage.
• Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
• Collaborate with the data engineering team to integrate crawled data into pipelines.
• Stay updated with the latest tools, techniques, and best practices in web crawling.

Education

Bachelor's degree in Computer Science / Information Technology

Tools

Scrapy

Requests

BeautifulSoup

Selenium

Proxy rotation

Job description

Job Description:

Job Title: Web Crawling Engineer

Experience:3-5 Years

Location: Hyderabad

We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

• Design, develop, and maintain scalable and efficient web crawlers for data extraction.
• Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
• Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
• Implement and manage proxy rotation and session management strategies.
• Ensure high-quality and structured data extraction, cleaning, and storage.
• Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
• Collaborate with the data engineering team to integrate crawled data into pipelines.
• Stay updated with the latest tools, techniques, and best practices in web crawling.

Required Skills & Qualifications

• Strong programming skills in Python.
• Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
• Hands-on experience solving blocking issues, handling
• Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
• Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
• Experience with databases (SQL/NoSQL) for storing crawled data.
• Strong debugging, problem-solving, and analytical skills.

Good to Have

• Experience with cloud environments (AWS, GCP, Azure).
• Knowledge of data pipelines and ETL processes.

Education & Experience

• Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience.
• 3+ years of experience in web crawling or related fields