AI Engineer — Research Agents (Full-Stack)

Socket.dev; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Socket.dev is hiring to design and ship agentic systems that fetch, extract, and normalize data across the web and APIs. You will build and operate search pipelines on OpenSearch/Elasticsearch and own robust web scraping with anti-bot tactics.

You will develop backend services in Python + FastAPI, scale workloads on AWS + Temporal, and integrate external APIs for enrichment and caching. Strong data wrangling with Pandas is essential.

Qualifications

• Hands-on experience with agentic architectures and prompt engineering.
• Deep knowledge of OpenSearch/Elasticsearch: index design, ingestion, migrations.
• Proven web scraping expertise including CAPTCHA handling and anti-bot flows.
• AWS + Temporal in production (two or more services).
• Building high-throughput data/IO pipelines with concurrency and retries.
• Integrating diverse external APIs and stable interfaces.
• Strong data wrangling with Pandas or equivalents.
• Familiarity with Stripe and basic front-end changes.

Responsibilities

• Design and ship agentic systems with tool calling and structured outputs.
• Build and operate search/indexing pipelines on OpenSearch/Elasticsearch.
• Own robust web scraping, including CAPTCHA handling and proxies.
• Develop backend services in Python + FastAPI with observability.
• Scale workloads on AWS + Temporal with autoscaling and fault tolerance.
• Parallelize external API requests with safe rate limiting and retries.
• Integrate third-party APIs for enrichment and caching.
• Transform data with Pandas for QA and reporting.
• Contribute to billing and occasional frontend changes.

Skills

Agentic architectures

Prompt engineering

Web scraping

Python

Async IO

Debugging

Product ownership

Tools

OpenSearch/Elasticsearch

AWS

Temporal

React

TypeScript

SQS/SNS

Lambda

Playwright/Selenium

Redis

Postgres

Terraform

Job description

What you’ll do

• Design and ship agentic systems (tool calling, multi-agent workflows, structured outputs) that reliably fetch, extract, and normalize data across the web and APIs.
• Build and operate search/indexing pipelines on OpenSearch/Elasticsearch (schema design, analyzers, reindex/data migration strategies, relevance tuning).
• Own robust web scraping: directory crawling, CAPTCHA handling, headless browsers, rotating proxies, anti-bot evasion, and backoff/retry policies.
• Develop backend services in Python + FastAPI with clean contracts and strong observability.
• Scale workloads on AWS + Temporal (batch/queue workers, autoscaling, fault tolerance, cost control).
• Parallelize external API requests safely (rate limits, idempotency, circuit breakers, retries, dedupe).
• Integrate third‑party APIs for enrichment and search; model and cache responses; manage schema evolution.
• Transform and analyze data using Pandas (or similar) for normalization, QA, and reporting.
• Pitch in across the stack: billing (Stripe), and occasional front‑end changes to ship end‑to‑end features.

Minimum requirements

• Hands-on experience with agentic architectures (tool calling, structured outputs/JSON, planning/execution loops) and prompt engineering.
• Deep knowledge of OpenSearch/Elasticsearch: index design, analyzers, ingestion pipelines, snapshots, rolling upgrades, and zero-downtime reindexing/data migrations.
• Proven web scraping expertise: solving CAPTCHAs, session/auth flows, proxy rotation, stealth techniques, and legal/ethical constraints.
• AWS + Temporal in production (at least two of: ECS/EKS, Lambda, SQS/SNS, Batch, Step Functions, CloudWatch).
• Building high-throughput data/IO pipelines with concurrency (asyncio/multiprocessing), resilient retries, and rate‑limit aware scheduling.
• Integrating diverse external APIs (auth patterns, pagination, webhooks); designing stable interfaces and backfills.
• Strong data wrangling with Pandas or equivalent; comfort with large CSV/Parquet workflows and memory/perf tuning.
• Familiarity with Stripe (subscriptions, metered billing, webhooks) and basic front‑end changes (React/TypeScript or similar).
• Excellent ownership, product sense, and pragmatic debugging.

Nice to have

• Entity resolution/record linkage at scale (probabilistic matching, blocking, deduping).
• Experience with Langfuse, OpenTelemetry, or similar for tracing/evals; task queues (Celery/RQ), Redis, Postgres.
• Search relevance (BM25/vector/hybrid), embeddings, and retrieval pipelines.
• Playwright/Selenium, stealth browsers, anti‑bot frameworks, CAPTCHA providers.
• CI/CD, infrastructure as code (Terraform), and cost/perf observability.
• Security & compliance basics for data handling and PII.