Full-Stack Web Scraping Engineer (Python & JavaScript – 3 to 8 Years)

FullStackTechies; Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

FullStackTechies in Bengaluru is seeking a skilled web scraper developer to design, develop, and maintain efficient web scraping systems. You will leverage advanced technologies, including AI and ML, to enhance the performance and accuracy of data extraction workflows.

The ideal candidate has a strong background in Python, familiarity with frameworks like Scrapy, and experience in large-scale cleaning and parsing of data. This role involves collaboration with data teams and implementing best practices for maintaining scrapers.

Qualifications

• Strong programming skills in Python (Scrapy, Requests, Playwright/Selenium).
• Experience with large-scale scraping systems.
• Familiarity with databases (SQL/NoSQL) for storing data.

Responsibilities

• Design, develop, and maintain large-scale web scrapers.
• Optimize scraping pipelines for scalability and performance.
• Monitor, debug, and maintain scrapers.

Skills

Web scraping expertise

Python proficiency

Experience with Scrapy

AI-driven techniques

Problem-solving skills

Education

Bachelor’s degree in computer science, IT, or related field

Tools

Scrapy

Requests

Playwright

Selenium

Docker

Kubernetes

Job description

Job Description:

What We Are Looking For:

• Strong expertise in web scraping with hands‑on experience in large‑scale data collection systems.
• Proficient in Python, including Scrapy, Requests, Playwright, and Selenium.
• Capable of designing, building, and maintaining robust and scalable scrapers for a variety of websites.
• Experience with AI‑driven techniques to improve scraping efficiency, data extraction accuracy, and automation workflows.
• Passionate about leveraging advanced technologies to solve complex data challenges.

Responsibilities:

• Design, develop, and maintain large‑scale web scrapers to collect structured and unstructured data from multiple sources.
• Optimize scraping pipelines for scalability, reliability, and performance.
• Work with Scrapy and related Python frameworks/libraries for crawler development.
• Monitor, debug, and maintain scrapers, ensuring high availability and fault tolerance.
• Implement AI/ML‑based solutions in scraping workflows, such as:
• Using LLMs for content parsing/cleaning.
• Automating CAPTCHA solving and anti‑bot evasion.
• Improving data matching, deduplication, and entity recognition.
• Collaborate with data engineers and product teams to ensure scraped data quality and usability.
• Maintain proper documentation, testing, and CI/CD practices for scraper code.

Qualifications:

• Bachelor’s degree in computer science, IT, or related field.
• Strong programming skills in Python (Scrapy, Requests, Playwright/Selenium).
• Experience with large‑scale scraping systems (distributed crawling, proxy rotation, request throttling).
• Solid understanding of web technologies (HTTP, HTML, CSS, JSON, AJAX).
• Experience working with databases (SQL/NoSQL) for storing and processing scraped data.
• Hands‑on experience in AI/ML tools (e.g., OpenAI, Hugging Face, spaCy, Transformers, or LangChain) applied to data extraction/processing.
• Familiarity with containerization and orchestration (Docker, Kubernetes) for scraper deployments.
• Strong problem‑solving skills for dealing with anti‑bot measures and rate limits.
• Excellent communication and collaboration skills.