Senior Python Developer - Web Scraping & Data Crawling

Globaldata; Hyderabad; On-site; INR 1,000,000 - 2,000,000; Full time

Job summary

Globaldata in Hyderabad is seeking a seasoned Senior Python Developer to own end-to-end data extraction, web crawling, and automated data pipelines. You will maintain full-stack crawling apps, optimize uptime, and architect scalable crawlers with anti-bot navigation strategies.

The role demands 5+ years in data crawling, strong Python and SQL skills, and hands-on experience with Scrapy, Selenium, and NLP exposure.

Qualifications

• 5+ years leading data crawling projects.
• Expertise in Python automation and web scraping.
• Hands-on with Scrapy, Requests, BeautifulSoup, and browser automation.

Responsibilities

• Architect, deploy, and scale data crawlers for large datasets.
• Maintain full-stack crawling apps with high uptime and data integrity.
• Design Python automation scripts to monitor pipelines and handle errors.
• Clean and structure unstructured web data using SQL and Python.
• Implement proxy management, CAPTCHA solving, and user-agent rotation.

Skills

Python

Python Automation Scripting

Data Crawling

Web Scraping

SQL

Tools

Scrapy

Requests

BeautifulSoup

Selenium

Playwright

Job description

Job Description

We are looking for a seasoned Senior Python Developer with a proven track record in Data Crawling and Web Scraping. In this role, you will own the end-to-end data extraction lifecycle, maintain full-stack web crawling applications, and build automated data pipelines.

If you are an analytical thinker who thrives on transforming raw web data into structured business intelligence, we want to hear from you.

Role & responsibilities

• End-to-End Crawling: Architect, deploy, and scale robust data crawlers to extract massive datasets from complex web architectures.


• Application Management: Maintain and optimize the existing full-stack web crawling application to ensure high uptime and data integrity.


• Pipeline Automation: Design and execute Python automation scripts to monitor pipelines, handle errors, and streamline workflows.


• Data Engineering: Clean, manipulate, and structure unstructured web data using SQL and Python libraries for downstream analytics.


• Anti-Bot Navigation: Implement advanced strategies to bypass anti-scraping mechanisms, including proxy management, CAPTCHA solving, and user-agent rotation.

Requirements & Technical Skills

• Core Language: Strong expertise in Python and Python Automation Scripting.


• Scraping Tech Stack: Advanced hands-on experience with Scrapy, Requests, BeautifulSoup, and browser automation tools (Selenium/Playwright).


• Databases: Proficient in SQL for data manipulation, storage, and retrieval.


• Core Experience: Mandatory hands-on experience leading data crawling projects for 5+ years.


• Soft Skills: Excellent analytical capabilities and strong professional communication skills.

Preferred candidate profile

• Exposure to NLP (Natural Language Processing) and NLTK.


• Basic understanding of Machine Learning (ML) data preprocessing workflows.

Thanks & Regards

Tejaswini.P

(Human Resource)