Lead Data Ingestion & Scraping Systems Architect

Jobgether SRL; United States; Remote; USD 110.000 - 150.000; Vollzeit

Zusammenfassung

Jobgether is seeking a seasoned data engineer to design and scale high-performance web scraping and data ingestion systems in the United States. You will work at the intersection of data engineering, automation, and AI infrastructure, leading the development of scalable collection platforms.

Ideal candidates have 4+ years in web scraping or data engineering, deep knowledge of anti-bot measures (Cloudflare, Akamai), and strong Python or Go skills with distributed task queues.

Qualifikationen

• 4+ years in web scraping or data engineering.
• Experience with anti-bot systems (Cloudflare, Akamai).
• Proficient in Python or Go and distributed task queues.

Aufgaben

• Lead development of scalable data collection systems and reliable ingestion.
• Build fault-tolerant scraping pipelines for Instagram, TikTok, YouTube, and X.
• Architect storage with Parquet, Iceberg, Snowflake, and S3.

Kenntnisse

Web scraping

Data engineering

Reverse engineering

Anti-bot awareness

Tools

Python

Go

Distributed task queues

Parquet

Iceberg

Snowflake

S3

Cloudflare

Akamai

Jobbeschreibung

Jobgether is seeking a seasoned data engineer to design and scale high-performance web scraping and data ingestion systems in the United States. You will work at the intersection of data engineering, automation, and AI infrastructure, leading the development of scalable collection platforms.

Ideal candidates have 4+ years in web scraping or data engineering, deep knowledge of anti-bot measures (Cloudflare, Akamai), and strong Python or Go skills with distributed task queues.