Senior Scraping Engineer — Scalable Data Pipelines

NielsenIQ; United States; Teletrabalho; USD 120 000 - 180 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Flexible working environment

Volunteer time off

LinkedIn Learning

Employee-Assistance-Program (EAP)

Resumo da oferta

NIQ is seeking a seasoned scraping engineer to design and implement scalable data extraction pipelines across web and mobile sources. You will join a global team of engineers and data scientists, driving innovations in anti-bot techniques, proxy management, and large-scale data collection.

You will collaborate with Customer Success, Operations, and Data teams to deliver high-quality data quickly, while maintaining robust monitoring and fault-tolerance in distributed systems.

Qualificações

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies and how to bypass them.
• Experience with Scrapy, Requests, httpx, Selenium, Playwright.
• Familiarity with structured storage formats (Parquet, Avro) and databases like MongoDB/Redis.
• Experience with Linux and cloud environments (AWS, GCP).

Responsabilidades

• Design and build distributed scraping architectures that can scale across thousands of domains.
• Reverse engineer APIs and mobile apps using tools like Frida, mitmproxy, Charles Proxy, burp and many more.
• Detect common anti-bot measures to build crawlers that appear more human-like.
• Lead technical innovation in scraping: prototype new approaches and share knowledge with the team.
• Set up anomaly detection and monitoring to detect blockers or failures early.

Conhecimentos

Python

Large-scale scraping

Anti-bot tech

Debugging

Linux

Kubernetes basics

Proxy management

Data storage formats

MongoDB

Redis

Ferramentas

Scrapy

Selenium

Playwright

mitmproxy

Charles Proxy

burp

Frida

httpx

Descrição da oferta de emprego

NIQ is seeking a seasoned scraping engineer to design and implement scalable data extraction pipelines across web and mobile sources. You will join a global team of engineers and data scientists, driving innovations in anti-bot techniques, proxy management, and large-scale data collection.

You will collaborate with Customer Success, Operations, and Data teams to deliver high-quality data quickly, while maintaining robust monitoring and fault-tolerance in distributed systems.