Remote Senior Data Infrastructure Engineer - Scraping & Scale

Jobgether SRL; Australia; Remote; AUD 140,000 - 190,000; Full time

Benefits offered by this job

Fully remote

Full-time employment

Job summary

Jobgether SRL represents a partner company seeking a Senior Data Infrastructure Engineer (Scraping & Scale) based in Australia. The role involves designing and scaling high-performance web scraping and data ingestion systems to handle millions of profiles, posts, and video records daily, with a strong emphasis on anti-bot resilience and reliability.

You will lead development of distributed pipelines, orchestration, and storage strategies while supporting AI/analytics workflows.

Qualifications

• Proven experience in high-volume web scraping and data engineering.
• Experience handling anti-bot systems (Cloudflare, Akamai, PerimeterX).
• Strong programming skills in Python or Go.
• Experience with browser automation and scraping frameworks (Playwright, Puppeteer, Scrapy, Selenium).
• Knowledge of distributed systems and task queues (Temporal, Ray, Kafka, Redis).
• Strong SQL skills and data lake/warehouse familiarity.

Responsibilities

• Design and operate scalable web scraping infrastructure with resistance to blocks.
• Build high-throughput scraping pipelines for multiple platforms (Instagram, TikTok, YouTube, X).
• Develop distributed queues and orchestration with Temporal, Kafka, Ray, Celery.
• Plan storage for structured/unstructured data using Parquet, Iceberg, Snowflake, S3.
• Implement monitoring, alerts, and recovery for outages and platform changes.
• Ensure reliable ingestion at scale for daily millions of records.
• Enable analytics/AI workloads via robust data infrastructure.

Skills

Python

Go

Playwright

Puppeteer

Scrapy

Selenium

Temporal

Ray

Kafka

Redis

SQL

AWS S3

Parquet

Education

Bachelor's degree in CS/Engineering

Tools

Snowflake

Apache Iceberg

Kubernetes

Twilio/Proxy networks

Job description

Jobgether SRL represents a partner company seeking a Senior Data Infrastructure Engineer (Scraping & Scale) based in Australia. The role involves designing and scaling high-performance web scraping and data ingestion systems to handle millions of profiles, posts, and video records daily, with a strong emphasis on anti-bot resilience and reliability.

You will lead development of distributed pipelines, orchestration, and storage strategies while supporting AI/analytics workflows.