Senior Consultant - Web Scraping

Keka Technologies Private Limited; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Chryselys is seeking a senior data scraping engineer to own and extend a production-bound POC that integrates resilient web crawling with an AWS Bedrock-based verifier in a healthcare-data context.

You will replace SERP scraping with paid APIs, optimize per-domain concurrency, implement proxy rotation, and enhance observability with structured logs, metrics, and alerts. Strong emphasis on robots.txt compliance and licensing records.

Qualifications

• 8+ years in data acquisition and 5+ years owning a scraping platform end-to-end.
• Proven scale: 1,000+ domains or 10M+ pages/month in production.
• Experience modernizing legacy scrapers with reliability/cost improvements.
• Mentored engineers; set crawl standards and on-call playbooks.

Responsibilities

• Replace scraped SERPs with a paid search API behind the existing search_domain interface.
• Move fetch.py to async httpx with per-domain concurrency limits and politesse budgets.
• Introduce proxy rotation and egress management; retire the single-IP failure mode.
• Make failure loud with typed outcomes, structured logs, metrics, drift and volume alerts.
• Reverse-engineer payer XHR/JSON endpoints to replace Playwright recipes where possible.
• Honor robots.txt Disallow and Crawl-delay; build per-domain ToS and licensing register.
• Containerize and schedule the pipeline; add CI running offline tests on every change.
• Add HAR replay and golden-file parser tests on real payer HTML, plus daily canary crawls.

Skills

Data acquisition

Scraping

LLM verification

AWS Bedrock

Web crawling

Proxy rotation

CI/CD

Logging

Monitoring

Open to scale

Education

Equivalent practical experience

Tools

Scrapy

Playwright

Crawlee

Python

Job description

Chryselys is a Great Place to Work Certified, Pharma Analytics & Business consulting company that delivers data-driven insights leveraging AI-powered, cloud-native platforms to achieve high-impact transformations.

We specialize in digital technologies and advanced data science techniques that provide strategic and operational insights.

Who we are:

People - Our team of industry veterans, advisors and senior strategists have diverse backgrounds and have worked at top tier companies.

Quality - Our goal is to deliver the value of a big five consulting company without the big five cost.

Technology - Our solutions are Business centric built on cloud native technologies.

Role summary: Own and extend a production-bound POC that combines resilient web-crawling engineering with an AWS Bedrock-based LLM verification agent, in a healthcare-data-compliance-sensitive context.

Responsibilities

• Replace scraped SERPs with a paid search API behind the existing search_domain() interface.
• Move fetch.py to async httpx with per-domain concurrency limits and politeness budgets.
• Introduce proxy rotation and egress management; retire the single-IP failure mode.
• Make failure loud: typed outcomes, structured logs, metrics, drift and volume alerts.
• Reverse-engineer payer XHR/JSON endpoints to replace Playwright recipes wherever possible.
• Honor robots.txt Disallow and Crawl-delay; build a per-domain ToS and licensing register.
• Containerize and schedule the pipeline; add CI running the offline tests on every change.
• Add HAR replay and golden-file parser tests on real payer HTML, plus daily canary crawls.

Experience - Required

• 8+ years in data acquisition; 5+ years owning a scraping platform end to end.
• Proven scale: 1,000+ distinct domains or 10M+ pages/month sustained in production.
• Rescued a brittle legacy scraper, with before/after reliability and cost numbers.
• Mentored engineers; set crawl standards, review practice, and on-call runbooks.
• Degree optional — equivalent practical experience is fully accepted.

Nice-to-Have

• US payer policy, formulary, or prior-authorization document domain knowledge.
• Open-source contributions to Scrapy, Playwright, Crawlee, or a parsing library.
• LLM-assisted extraction at controlled cost — we run AWS Bedrock in verifier/.
• Compliance or legal-review exposure on data acquisition programmes.

Why Join Us?

• Work on Advanced Analytical techniques using AI/ML in the Market Access functions for the pharmaceutical domain with industry-leading tools.
• Collaborate with Global and US teams to advance Market Access Contracting and Patient Analytics.
• Opportunity to drive AI/ML-based innovations in Market Access Analytics.
• Competitive salary, learning opportunities, and a dynamic work environment

Chryselys is proud to be an Equal Employment Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.