Lead Web Scraping Engineer (Python, Scrapy)

NielsenIQ; Województwo mazowieckie; On-site; PLN 180,000 - 280,000; Full time

Benefits offered by this job

Flexible working environment

Volunteer time off

LinkedIn Learning

Employee-Assistance-Program (EAP)

Job summary

NIQ is seeking a seasoned scraping engineer to design and implement scalable distributed scraping systems. You will reverse engineer APIs and apps, bypass anti-bot measures, and innovate with the team to deliver high-quality data at scale.

You will collaborate with global teams, leverage tools like Frida, mitmproxy, and Playwright, and apply best practices for reliability and performance in a fast-paced environment.

Qualifications

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies and how to bypass them, fingerprinting, WAFs, JavaScript challenges, etc.
• Experience with Scrapy, Requests, httpx, Selenium, Playwright, or custom headless browser drivers.
• Skilled at handling proxies (residential, datacenter, rotating), session management, cookie injection, and TLS tweaking.
• Strong debugging skills across browser, HTTP traffic, and device-level interactions.
• Familiarity with Parquet, Avro, or real-time stream processing.
• Good knowledge of databases like MongoDB, InfluxDB, and Redis, including optimization.

Responsibilities

• Design and build distributed scraping architectures that can scale across thousands of domains.
• Reverse engineer APIs and mobile apps using tools like Frida, mitmproxy, Charles Proxy, burp and many more.
• Detect common anti-bot measures to build crawlers that appear more human-like.
• Lead technical innovation in scraping: identify edge cases, prototype new approaches, and share knowledge with the team.
• Set up anomaly detection and monitoring to detect blockers or failures early

Skills

Python

anti-bot technologies

Scrapy

Requests

httpx

Selenium

Playwright

proxies management

debugging

Parquet/Avro

MongoDB

Redis

Kubernetes basics

Linux/cloud

Tools

Frida

mitmproxy

Charles Proxy

Burp Suite

Job description

NIQ is seeking a seasoned scraping engineer to design and implement scalable distributed scraping systems. You will reverse engineer APIs and apps, bypass anti-bot measures, and innovate with the team to deliver high-quality data at scale.

You will collaborate with global teams, leverage tools like Frida, mitmproxy, and Playwright, and apply best practices for reliability and performance in a fast-paced environment.