Data Engineer(Web Crawling)

OSI Digital, Inc.; Hyderabad; On-site; INR 600,000 - 1,000,000; Full time

Job summary

OSI Digital, Inc. in Hyderabad is seeking a Web Crawling Engineer with 3–6 years of Python experience to design scalable crawlers and scrapers. You will tackle anti-blocking, proxy rotation, and captcha challenges while building robust data pipelines.

Responsibilities include implementing crawlers using Scrapy and Requests, maintaining data quality, and optimizing performance for reliability. Experience with SQL/NoSQL databases and cloud environments is a plus.

Qualifications

• Strong programming in Python with hands-on crawlers experience.
• Experience with crawling frameworks and proxy management.
• Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.

Responsibilities

• Design, develop and maintain scalable web crawlers for data extraction.
• Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
• Handle anti-scraping measures, including IP blocks, rate limits and captchas.
• Implement and manage proxy rotation and session management strategies.
• Ensure high-quality and structured data extraction, cleaning and storage.
• Monitor crawler performance, troubleshoot issues and optimise reliability and efficiency.
• Collaborate with data engineering to integrate crawled data into pipelines.
• Stay updated with latest tools, techniques in web crawling.

Skills

Python

HTTP/Networking basics

Proxy management

Anti-blocking techniques

Data extraction

Education

Bachelors in CS/IT or equivalent

Tools

Scrapy

Requests

BeautifulSoup

Selenium

Job description

Job Title: Web Crawling Engineer

Experience:3-6 Years

Notice :Immediate -15 Days

Interview Process: Two Rounds(One Round In Person Mandatory)

Location: Hyderabad

We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands‑on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti‑blocking techniques, and proxy management, along with experience solving challenges like captchas and rate‑limiting.

Key Responsibilities

• Design, develop and maintain scalable and efficient web crawlers for data extraction.
• Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
• Handle anti‑scraping measures, including IP blocks, rate limits and captchas.
• Implement and manage proxy rotation and session management strategies.
• Ensure high‑quality and structured data extraction, cleaning and storage.
• Monitor crawler performance, troubleshoot issues and optimise for reliability and efficiency.
• Collaborate with the data engineering team to integrate crawled data into pipelines.
• Stay updated with the latest tools, techniques and best practices in web crawling.

Required Skills & Qualifications

• Strong programming skills in Python.
• Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
• Hands‑on experience solving blocking issues, handling
• Good understanding of proxy usage, rotation and fingerprinting avoidance techniques.
• Knowledge of HTTP, cookies, headers, sessions and request/response cycles.
• Experience with databases (SQL/NoSQL) for storing crawled data.
• Strong debugging, problem‑solving and analytical skills.

Good to Have

• Experience with cloud environments (AWS, GCP, Azure).
• Knowledge of data pipelines and ETL processes.

Education & Experience

• Bachelors degree in Computer Science, Information Technology or equivalent practical experience.
• 3+ years of experience in web crawling or related fields