Distributed Web Crawler Engineer

Remotedxb; Dubai; On-site; AED 350,000 - 620,000; Full time

Job summary

Remotedxb in Dubai is seeking an experienced infrastructure engineer to design and maintain large-scale web crawlers and data pipelines. You will tackle high-throughput systems, handle anti-bot defenses, and optimize for performance and reliability.

You should be proficient in Go, Rust, Python, Java, or C++, familiar with HTTP, networking, and distributed systems, and able to work with big data and cloud or bare-metal infrastructure.

Qualifications

• Proficiency in multiple programming languages (Go, Rust, Python, Java, or C++).
• Experience building web crawlers or large-scale data pipelines.
• Solid understanding of HTTP, networking, and browser behavior.
• Familiarity with distributed systems and parallel processing.
• Experience with TB–PB scale data handling preferred.
• Ability to debug unstable or adversarial environments.

Responsibilities

• Build and maintain large-scale web crawlers across diverse domains.
• Design high-throughput, fault-tolerant systems for data collection.
• Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites.
• Develop pipelines for cleaning, deduplication, filtering, and normalization.
• Construct and maintain datasets for research and model training.
• Monitor crawl performance, coverage, and data quality.
• Collaborate with research teams to align data collection with modeling needs.
• Optimize infrastructure for cost, latency, and reliability.

Skills

Go

Rust

Python

Java

C++

Web crawlers

Data pipelines

HTTP networking

Distributed systems

Big data

Debugging in unstable envs

Tools

Playwright

Puppeteer

Proxy systems

IP rotation

Cloud/bare-metal infra

Job description

Remotedxb in Dubai is seeking an experienced infrastructure engineer to design and maintain large-scale web crawlers and data pipelines. You will tackle high-throughput systems, handle anti-bot defenses, and optimize for performance and reliability.

You should be proficient in Go, Rust, Python, Java, or C++, familiar with HTTP, networking, and distributed systems, and able to work with big data and cloud or bare-metal infrastructure.