Senior Web Data Pipeline Engineer – Resilient Crawling

ZEN; Warszawa; Presencial; PLN 120.000 - 160.000; Jornada completa

Descripción de la vacante

A financial tech company in Warsaw is looking for a Senior Web Data Acquisition Engineer to design and run high-throughput web data pipelines. The candidate should be proficient in web scraping, Python or Java/Kotlin, and have a strong understanding of data quality. You will work with a collaborative team using production-ready AI technologies, making decisions that balance accuracy and cost while ensuring robust system performance.

Formación

• Practical experience handling rate limits/challenges.
• Solid grounding in data quality: normalization, validation, deduplication, and schema/versioning.
• Experience in designing and building high-throughput data pipelines.

Responsabilidades

• Design, build, and run high-throughput web data pipelines.
• Make render vs. fetch decisions for accuracy, latency, and cost.
• Create robust extractors/parsers with auto-healing patterns.
• Engineer resilience against dynamic access controls.
• Own observability: success rate, latency, cost.

Conocimientos

Web data scraping

Python or Java/Kotlin

REST APIs

Data quality handling

Dynamic access control management

Descripción del empleo

A financial tech company in Warsaw is looking for a Senior Web Data Acquisition Engineer to design and run high-throughput web data pipelines. The candidate should be proficient in web scraping, Python or Java/Kotlin, and have a strong understanding of data quality. You will work with a collaborative team using production-ready AI technologies, making decisions that balance accuracy and cost while ensuring robust system performance.