Remote Data Infrastructure Engineer - Scraping & Scale

Jobgether SRL; United Kingdom; Remote; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Fully remote

Job summary

Jobgether SRL is seeking a Senior Data Infrastructure Engineer (Remote, UK) to design and scale high-performance web scraping and data ingestion systems. You will build core infrastructure to collect millions of profiles, posts, and video records daily while handling anti-bot challenges and distributed pipelines.

Work spans data engineering, automation, and AI infrastructure, aligning performance, reliability, and continuous improvement with business goals.

Qualifications

• At least 4 years of experience in high-volume web scraping or data engineering.
• Experience handling anti-bot systems such as Cloudflare, Akamai, or PerimeterX.
• Proficient in Python or Go and browser automation frameworks (Playwright/Scrapy/Selenium).
• Experience with distributed queues and task orchestration (Temporal, Ray, Kafka, Redis).
• Strong SQL skills and knowledge of data lakes/warehouses (Parquet/Iceberg/Snowflake/S3).
• Ability to design scalable, fault-tolerant data ingestion pipelines.

Responsibilities

• Design and manage web scraping infrastructure including residential proxies and session rotation.
• Build high-throughput scraping pipelines collecting data from multiple social platforms.
• Develop distributed queues and orchestration using Temporal, Ray, Kafka, and Celery.
• Architect storage using Parquet, Iceberg, Snowflake, and S3 for structured and unstructured data.
• Implement monitoring, alerts, and recovery for platform changes and proxy failures.
• Maintain reliable data ingestion at scale for daily millions of records.
• Enable downstream AI and analytics through robust data infrastructure.

Skills

Python

Go

Playwright

Puppeteer

Scrapy

Selenium

Temporal

Ray

Apache Kafka

Redis

SQL

Parquet

Iceberg

Snowflake

Amazon S3

Tools

Python

Go

Temporal

Ray

Apache Kafka

Redis

SQL

Parquet

Iceberg

Snowflake

Amazon S3

Job description

Jobgether SRL is seeking a Senior Data Infrastructure Engineer (Remote, UK) to design and scale high-performance web scraping and data ingestion systems. You will build core infrastructure to collect millions of profiles, posts, and video records daily while handling anti-bot challenges and distributed pipelines.

Work spans data engineering, automation, and AI infrastructure, aligning performance, reliability, and continuous improvement with business goals.