Senior Data Infrastructure Engineer - Remote Scraping

Jobgether SRL; Polska; Remote; PLN 210.000 - 320.000; Vollzeit

Benefits dieser Stelle

Fully remote work

Full-time employment

Zusammenfassung

Jobgether SRL is seeking a Senior Data Infrastructure Engineer (Scraping & Scale) based in Poland for a fully remote role. You will design and scale high-performance web scraping and data ingestion systems, collecting millions of profiles, posts, and videos daily.

The role emphasizes anti-bot challenges, resilient distributed pipelines, and enabling downstream AI workflows. Strong Python/Go, browser automation, and SQL/data warehouse skills are essential.

Qualifikationen

• At least 4 years of experience in high-volume web scraping, data engineering, or reverse engineering.
• Strong experience handling advanced anti-bot systems (Cloudflare, Akamai, PerimeterX) using TLS impersonation, browser automation and proxy mgmt.
• Advanced proficiency in Python or Go.
• Extensive experience with browser automation and scraping frameworks (Playwright, Puppeteer, Scrapy, Selenium).
• Practical knowledge of distributed systems and task queues (Temporal, Ray, Kafka, Redis).
• Strong SQL skills and experience with modern analytical data lakes and data warehouses.
• Ability to design scalable, fault-tolerant systems and troubleshoot data ingestion challenges.

Aufgaben

• Lead development and maintenance of scalable data collection systems for reliable ingestion.
• Design web scraping infrastructure with residential proxies, TLS fingerprinting, and session rotation.
• Build fault-tolerant, high-throughput scraping pipelines for major platforms (Instagram, TikTok, YouTube, X).
• Develop distributed queues and workflows using Temporal, Ray, Kafka, and Celery.
• Architect storage for structured and unstructured data (Parquet, Iceberg, Snowflake, S3).
• Implement monitoring, alerts, and recovery for platform changes and proxy failures.
• Ensure daily processing of millions of records and support AI/analytics workflows.

Kenntnisse

Web scraping

Anti-bot systems

Python/Go

Browser automation

Distributed systems

SQL / data warehouses

Data ingestion

Tools

Playwright

Puppeteer

Scrapy

Selenium

Temporal

Kafka

Redis

Jobbeschreibung

Jobgether SRL is seeking a Senior Data Infrastructure Engineer (Scraping & Scale) based in Poland for a fully remote role. You will design and scale high-performance web scraping and data ingestion systems, collecting millions of profiles, posts, and videos daily.

The role emphasizes anti-bot challenges, resilient distributed pipelines, and enabling downstream AI workflows. Strong Python/Go, browser automation, and SQL/data warehouse skills are essential.