Remote Data Infrastructure Engineer (Scraping & Scale)

Jobgether; United States; Remote; USD 65,000 - 98,000; Full time

Benefits offered by this job

Fully remote work arrangement

Full-time employment

Job summary

Jobgether, on behalf of a partner company, seeks a Senior Data Infrastructure Engineer (Scraping & Scale) for a fully remote role based in the United Arab Emirates.

You will design and scale high-performance web scraping and data ingestion systems, tackling anti-bot challenges and building resilient pipelines for millions of profiles daily. This role sits at the intersection of data engineering, automation, and AI infrastructure.

Qualifications

• 4+ years of experience in high-volume web scraping or data engineering.
• Experience with advanced anti-bot systems (Cloudflare, Akamai, PerimeterX).
• Proficient in Python or Go.
• Experience with browser automation and scraping frameworks (Playwright, Puppeteer, Scrapy, Selenium).
• Knowledge of distributed queues (Temporal, Ray, Kafka, Redis) and SQL data lakes/warehouses.

Responsibilities

• Design and maintain scalable web scraping infrastructure with proxy networks and session rotation.
• Build high-throughput pipelines to collect data from major platforms (Instagram, TikTok, YouTube, X).
• Develop distributed queues and orchestration with Temporal, Ray, Kafka, and Celery.
• Architect storage using Parquet, Iceberg, Snowflake, and S3; monitor and recover from proxy failures.
• Ensure reliable ingestion and support daily processing of millions of records for analytics and AI workflows.

Skills

Python

Go

Web scraping

Browser automation

Distributed systems

SQL

Kafka

Redis

Temporal

Ray

Tools

Playwright

Puppeteer

Scrapy

Selenium

Job description

Jobgether, on behalf of a partner company, seeks a Senior Data Infrastructure Engineer (Scraping & Scale) for a fully remote role based in the United Arab Emirates.

You will design and scale high-performance web scraping and data ingestion systems, tackling anti-bot challenges and building resilient pipelines for millions of profiles daily. This role sits at the intersection of data engineering, automation, and AI infrastructure.