Senior Data Infrastructure Engineer — Scalable Web Scraping (Remote)

Jobgether SRL; United Arab Emirates; Remote; AED 180,000 - 270,000; Full time

Benefits offered by this job

Fully remote work arrangement

Full-time employment

Job summary

Jobgether SRL, listed partner, seeks a Senior Data Infrastructure Engineer (Scraping & Scale) in the United Arab Emirates for a fully remote role. You will design and scale high-performance web scraping and data ingestion systems to collect millions of profiles and posts daily, tackling anti-bot challenges and building resilient pipelines.

This role blends data engineering, automation, and AI infrastructure, requiring deep technical expertise, reliability focus, and continuous improvement.

Qualifications

• At least 4 years of experience in high-volume web scraping, data engineering, or reverse engineering.
• Experience handling advanced anti-bot systems (Cloudflare, Akamai, PerimeterX) using TLS impersonation and proxy management.
• Proficient in Python or Go, plus browser automation (Playwright, Puppeteer, Scrapy, or Selenium).
• Experience with distributed systems and data storage technologies (Temporal, Ray, Kafka, Redis; Parquet, Iceberg, Snowflake, S3).

Responsibilities

• Design and maintain scalable web scraping infrastructure using residential proxies, TLS fingerprinting, and session rotation.
• Build fault-tolerant, high-throughput pipelines to collect data from multiple sources.
• Develop distributed queues and workflow orchestration with Temporal, Kafka, and Celery.
• Architect storage for structured and unstructured data with Parquet, Iceberg, Snowflake, and S3.
• Implement monitoring, alerts, and recovery for platform changes and proxy failures.
• Ensure reliable ingestion at scale and support daily processing of millions of records.
• Enable downstream AI and analytics workflows via robust data infrastructure.

Skills

Web scraping

Distributed systems

Data engineering

Cloud infrastructure

Tools

Python

Go

Playwright

Puppeteer

Scrapy

Selenium

Temporal

Ray

Apache Kafka

Celery

Redis

Parquet

Apache Iceberg

Snowflake

Amazon S3

Cloudflare

Akamai

PerimeterX

Job description

Jobgether SRL, listed partner, seeks a Senior Data Infrastructure Engineer (Scraping & Scale) in the United Arab Emirates for a fully remote role. You will design and scale high-performance web scraping and data ingestion systems to collect millions of profiles and posts daily, tackling anti-bot challenges and building resilient pipelines.

This role blends data engineering, automation, and AI infrastructure, requiring deep technical expertise, reliability focus, and continuous improvement.