Web Scraping Specialist

Social Fetch; United States; On-site; USD 120,000 - 170,000; Full time

Benefits offered by this job

Flexible hours

Unlimited PTO

Equipment stipend

Learning budget

Health benefits

Job summary

Social Fetch is building a unified REST API from public social data. You’ll own the fetch layer, researching platforms, implementing scrapers, and hardening them for production across TikTok, Instagram, YouTube, Reddit, LinkedIn, Facebook, and more.

Your role focuses on reliable data extraction, handling anti-bot challenges, and maintaining stable field mappings as platforms evolve. This is a fully remote, async environment with flexible hours and a strong emphasis on collaboration and

Qualifications

• Several years of hands-on web scraping experience.
• Familiarity with anti-bot systems and browser automation.
• Experience parsing messy HTML and JSON data.
• Proficient in TypeScript/JavaScript and production debugging.

Responsibilities

• Build and maintain scrapers and integrations for major social platforms — profiles, posts, search, media, comments, engagement metrics, and platform‑specific endpoints.
• Diagnose fetch failures: HTTP errors, schema drift, rate limits, bot detection, geo blocks, CAPTCHAs, and responses that look fine but parse to garbage.
• Design fetch strategies that survive bad days — retries with backoff, fallbacks between mobile and web APIs, session handling, proxy rotation when it’s worth the cost.
• Keep normalized API field mappings stable when platforms rename or nest fields differently.
• Write internal notes on platform quirks (auth flows, pagination traps, fields that lie) so the next person doesn’t start from zero.
• Watch production quality signals — error rates, empty fields, latency spikes — and fix regressions before customers open tickets.
• Review support escalations that smell like platform drift, not user error.

Skills

Web scraping experience

Playwright/Puppeteer

HTML/JSON parsing

Production debugging

TypeScript/JavaScript

Independent remote work

Education

Bachelor’s degree

Tools

Playwright

Puppeteer

HTTP clients

Proxy rotation

Observability tools

Effect-TS

Job description

About The Role

Social Fetch normalises public social data into a single REST API. Our customers integrate once and read followerCount whether the underlying platform called it something else. When Instagram changes a field name or TikTok rotates an anti-bot challenge, they shouldn’t find out from a broken cron job.

You’ll own the fetch layer — researching how platforms serve data, implementing and maintaining scrapers, and hardening them when things break in production. That covers profiles, posts, search, comments, transcripts, and ad‑library lookups across TikTok, Instagram, YouTube, Reddit, LinkedIn, Facebook, and more. The stack is TypeScript throughout: monorepo, Effect for the fetch layer’s control flow, Playwright where we need a browser, plain HTTP where we don’t.

Team & how we work

You’ll work with the engineers who built the API and support who see failures first. When a platform drifts, support often spots the pattern before monitoring — Slack threads with requestIds, sample URLs, and "this worked yesterday" context.

Fully remote, flexible hours. Incidents don’t respect timezones, so sometimes an urgent fix lands outside your usual window — day‑to‑day work is async. Write up what you found, open a PR, leave notes. No dedicated SRE team; reliability is everyone’s job, and yours most of all.

What you’ll do

• Build and maintain scrapers and integrations for major social platforms — profiles, posts, search, media, comments, engagement metrics, and platform‑specific endpoints.
• Diagnose fetch failures: HTTP errors, schema drift, rate limits, bot detection, geo blocks, CAPTCHAs, and responses that look fine but parse to garbage.
• Design fetch strategies that survive bad days — retries with backoff, fallbacks between mobile and web APIs, session handling, proxy rotation when it’s worth the cost.
• Keep normalized API field mappings stable when platforms rename or nest fields differently.
• Write internal notes on platform quirks (auth flows, pagination traps, fields that lie) so the next person doesn’t start from zero.
• Watch production quality signals — error rates, empty fields, latency spikes — and fix regressions before customers open tickets.
• Review support escalations that smell like platform drift, not user error.

What we’re looking for

• Several years of hands‑on web scraping experience. Tutorial‑level Playwright scripts aren’t enough — we need someone who’s been paged when a site changed its HTML.
• Real familiarity with anti‑bot systems, browser automation (Playwright/Puppeteer), HTTP clients, and parsing messy HTML and JSON (including data hidden in script tags).
• Comfort with production debugging: logs, reproducing a failing URL, shipping a fix, watching the error rate drop.
• Strong TypeScript or JavaScript — our codebase is TypeScript and you’ll be in it daily.
• Pragmatic instincts about public data, ToS grey areas, and when to stop digging.
• Ability to work independently in a remote, async team — you unblock yourself, document what you learned, and ask for help when you’re stuck.

Nice to have

• Experience maintaining scrapers across many platforms, not just one site you know inside out.
• Familiarity with proxy providers, CAPTCHA services, and reverse‑engineering mobile or web API calls.
• Prior work at a data or API product company where uptime mattered to paying customers.
• Exposure to observability tooling — structured logs, request IDs, error tracking.
• Experience with Effect (Effect-TS) or similar typed‑effect/functional TypeScript libraries.

Benefits

Full-time team members receive:

• Flexible hours — Async‑first. Ship your work; we don’t track screen time or mandate overlap windows unless a role specifically needs it.
• Unlimited PTO — Take the time you need. Give the team a heads‑up for longer breaks so support and on‑call handoffs stay covered.
• Equipment stipend — Home office setup — desk, monitor, chair, whatever actually makes you productive.
• Learning budget — Courses, books, and conferences tied to your role. Scraping conferences, writing workshops, support tooling — all fair game.
• Health benefits — Health coverage for full‑time team members. Details and eligibility depend on location; we walk through it during hiring.