Senior Python Developer

Ninestars Information Technologies; Bengaluru; On-site; INR 700,000 - 1,100,000; Full time

Job summary

Ninestars Information Technologies is seeking a skilled web scraping engineer to design and develop scalable scraping solutions for both structured and unstructured websites. You will build robust crawlers using Scrapy, Playwright, Selenium, and BeautifulSoup, and develop browser automation scripts for dynamic JavaScript-based sites.

You will integrate scraped data with databases and APIs, handle pagination, authentication, cookies, sessions, anti-bot measures, and optimize performance via

Qualifications

• Solid experience building web scrapers for structured and unstructured sites.
• Proficient with Scrapy, Playwright, Selenium, and BeautifulSoup.
• Experience with browser automation and dynamic JS websites.
• Ability to integrate scraped data with databases and APIs.
• Familiarity with pagination, authentication, cookies, sessions, and anti-bot measures.
• Knowledge of data quality, validation, and transformation.

Responsibilities

• Design and develop scalable web scraping solutions for structured and unstructured websites.
• Build robust crawlers using Scrapy, Playwright, Selenium, and BeautifulSoup.
• Develop browser automation scripts for dynamic JavaScript-based websites.
• Integrate scraped data with databases, APIs, and downstream applications.
• Handle pagination, authentication, cookies, sessions, anti-bot, proxy and rate limiting.
• Optimize scraping performance using asynchronous programming and parallel processing.
• Monitor scraper health, troubleshoot failures, and implement retry mechanisms.
• Ensure data quality through validation, deduplication, and transformation.
• Collaborate with dev team and customer to deliver data.

Skills

Web crawling

DOM

JSON

Tools

Python

Playwright

HTML

CSS

SQL

WebDriver

Selenium

Scrapy

BeautifulSoup

Job description

Job Description:

Mandatory Skills: Python, Playwright, HTML, CSS, SQL, Web crawling, DOM, JSON, JavaScript, WebDriver

Nice-to-Have Skills: Selenium, Scrapy, BeautifulSoup, RegEx, Session/Cookie Management

Roles & Responsibility

• Design and develop scalable web scraping solutions for structured and unstructured websites.
• Build robust crawlers using Scrapy, Playwright, Selenium, and BeautifulSoup.
• Develop browser automation scripts for dynamic JavaScript-based websites.
• Integrate scraped data with databases, APIs, and downstream applications.
• Handle pagination, authentication, cookies, sessions, anti-bot, proxy and rate limiting.
• Optimize scraping performance using asynchronous programming and parallel processing.
• Monitor scraper health, troubleshoot failures, and implement retry mechanisms.
• Ensure data quality through validation, deduplication, and transformation.
• Collaborate with dev team and customer to deliver data.