Senior Web Data Pipeline Engineer – Resilient Crawling

ZEN; Warszawa; On-site; PLN 120,000 - 160,000; Full time

Job summary

A financial tech company in Warsaw is looking for a Senior Web Data Acquisition Engineer to design and run high-throughput web data pipelines. The candidate should be proficient in web scraping, Python or Java/Kotlin, and have a strong understanding of data quality. You will work with a collaborative team using production-ready AI technologies, making decisions that balance accuracy and cost while ensuring robust system performance.

Qualifications

• Practical experience handling rate limits/challenges.
• Solid grounding in data quality: normalization, validation, deduplication, and schema/versioning.
• Experience in designing and building high-throughput data pipelines.

Responsibilities

• Design, build, and run high-throughput web data pipelines.
• Make render vs. fetch decisions for accuracy, latency, and cost.
• Create robust extractors/parsers with auto-healing patterns.
• Engineer resilience against dynamic access controls.
• Own observability: success rate, latency, cost.

Skills

Web data scraping

Python or Java/Kotlin

REST APIs

Data quality handling

Dynamic access control management

Job description

A financial tech company in Warsaw is looking for a Senior Web Data Acquisition Engineer to design and run high-throughput web data pipelines. The candidate should be proficient in web scraping, Python or Java/Kotlin, and have a strong understanding of data quality. You will work with a collaborative team using production-ready AI technologies, making decisions that balance accuracy and cost while ensuring robust system performance.