Senior Data Engineer Web Scraping

Oxford Data Plan; Indore District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Oxford Data Plan in Indore is seeking an experienced Data Engineer to own web-scraping and web-data ingestion within our data platform. You will scale and stabilize scraping workloads and integrate them into our lakehouse architecture.

This senior, hands-on role requires building production scrapers, designing reliable patterns, and collaborating with a remote engineering team. You will apply AI tooling to improve development while ensuring compliance and robust operations.

Qualifications

• Production scraping systems experience at scale is required.
• Ability to own scraping projects from investigation to production.
• Strong Python engineering skills focused on maintainable apps.
• Experience with requests/httpx, BeautifulSoup, Scrapy, Playwright or Selenium.
• Understanding of HTTP/HTML/APIs and browser behavior.
• Experience handling pagination, authentication, sessions, and rate limits.
• Knowledge of data pipelines, data quality, storage and consumption downstream.
• Experience deploying and monitoring in cloud environments.
• Good debugging and independent problem-solving skills.
• Experience collaborating remotely with code reviews and documentation.

Responsibilities

• Own the development and operation of web-scraping and web-data ingestion pipelines.
• Design and establish a scalable web-scraping framework, including common patterns for extraction, scheduling, storage, monitoring, validation and failure handling.
• Design, build and maintain reliable production scrapers for new and existing data sources.
• Evaluate and make build-versus-buy recommendations for scraping infrastructure and third-party services, considering capability, reliability, cost, operational complexity and risk.
• Investigate websites and determine the most appropriate acquisition approach, including APIs, direct HTTP requests, HTML parsing, browser automation or third-party tooling.
• Ensure scraping activities appropriately consider internal policies, website terms, robots.txt, access restrictions, privacy and intellectual-property constraints, escalating unclear cases where needed.
• Diagnose and resolve issues caused by changing websites, dynamic content, authentication, rate limits and other operational challenges.
• Use AI tooling to work more effectively while understanding, reviewing and being able to defend the code you ship.

Skills

Python engineering

Web scraping at scale

Independent ownership

HTTP/HTML/APIs

Browser automation

Cloud deployment

Data pipelines

Debugging/solving

Code review/documentation

Remote collaboration

Tools

BeautifulSoup

Scrapy

Playwright

Selenium

Requests/httpx

Iceberg lakehouse

PySpark

Docker

Terraform

Grafana

Job description

Job Description:

We are looking for an experienced, self-driven Data Engineer to take ownership of web scraping and web-based data acquisition within our data platform.

We already use web scraping across a number of ingestion processes, and we are looking to improve the maturity, scale and robustness of this capability. This includes integrating scraping workloads into our lakehouse architecture, improving scheduling, monitoring and storage, and establishing reusable patterns for building and operating scrapers reliably.

This is a senior, hands-on role. We are looking for someone who has previously built and operated production web-scraping systems at scale and can independently take a new data source from investigation through implementation, deployment and ongoing support.

Roles and Responsibilities

• Own the development and operation of web-scraping and web-data ingestion pipelines.
• Design and establish a scalable web-scraping framework, including common patterns for extraction, scheduling, storage, monitoring, validation and failure handling.
• Design, build and maintain reliable production scrapers for new and existing data sources.
• Evaluate and make build-versus-buy recommendations for scraping infrastructure and third-party services, considering capability, reliability, cost, operational complexity and risk.
• Investigate websites and determine the most appropriate acquisition approach, including APIs, direct HTTP requests, HTML parsing, browser automation or third-party tooling.
• Ensure scraping activities appropriately consider internal policies, website terms, robots.txt, access restrictions, privacy and intellectual-property constraints, escalating unclear cases where needed.
• Diagnose and resolve issues caused by changing websites, dynamic content, authentication, rate limits and other operational challenges.
• Use AI tooling to work more effectively while understanding, reviewing and being able to defend the code you ship.

Required Qualifications

• Demonstrated professional experience building and operating production web-scraping systems at scale.
• Proven ability to independently own substantial scraping projects from initial investigation through production operation.
• Strong production Python engineering skills, including building maintainable applications rather than standalone scripts.
• Strong practical experience with scraping technologies such as Requests/httpx, BeautifulSoup, Scrapy, Playwright or Selenium.
• Good understanding of HTTP, HTML, APIs, JavaScript-rendered websites and browser/network behaviour.
• Experience handling common scraping challenges such as pagination, authentication, sessions, retries, rate limiting, concurrency and proxies.
• Strong understanding of data pipelines, data quality and how collected data should be validated, stored and consumed downstream.
• Experience deploying, monitoring and supporting production workloads in a cloud environment.
• Strong debugging and problem-solving skills, with the ability to work independently and make sensible engineering decisions.
• Experience working effectively within a remote engineering team, including code review, documentation and ticket-based workflows.

Desirable Skills

• Experience with AWS.
• Experience with lakehouse or data-lake architectures, particularly Iceberg.
• Experience with PySpark or other distributed data-processing technologies.
• Experience with Docker and containerised workloads.
• Experience with Terraform or other infrastructure-as-code tooling.
• Experience with Grafana or similar observability platforms.
• Experience operating high-volume or distributed crawling systems.
• Experience evaluating or operating commercial scraping, proxy or browser-infrastructure services.
• Experience implementing automated scraper testing, canary runs or source-drift detection.
• Experience working with legal, compliance, privacy or data-governance teams on web-data acquisition.