AI Engineer: Agentic Systems & Scalable Data Pipelines

Sixtyfour; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Sixtyfour is seeking an experienced Backend/Data Engineer to design agentic systems and scalable data pipelines in aSF-based environment. You will build search pipelines with OpenSearch, manage robust web scraping, and integrate third-party APIs for enrichment and search.

Strong Python+FastAPI, AWS, and prompt engineering experience are essential. Responsibilities include scaling workloads, handling CAPTCHA challenges, and ensuring fault-tolerant, cost-conscious operations across batch/streaming

Qualifications

• ,

Responsibilities

• Design and ship agentic systems (tool calling, multi-agent workflows, structured outputs) that fetch, extract, and normalize data across the web and APIs.
• Build and operate search/indexing pipelines on OpenSearch/Elasticsearch (schema design, analyzers, reindex/data migration strategies, relevance tuning).
• Own robust web scraping: directory crawling, CAPTCHA handling, headless browsers, rotating proxies, anti-bot evasion, and backoff/retry policies.
• Develop backend services in Python + FastAPI with clean contracts and strong observability.
• Scale workloads on AWS + Temporal (batch/queue workers, autoscaling, fault tolerance, cost control).
• Parallelize external API requests safely (rate limits, idempotency, circuit breakers, retries, dedupe).
• Integrate third-party APIs for enrichment and search; model and cache responses; manage schema evolution.
• Transform and analyze data using Pandas (or similar) for normalization, QA, and reporting.
• Pitch in across the stack: billing (Stripe), and occasional front-end changes to ship end-to-end features.

Skills

Agentic architectures

OpenSearch/Elasticsearch

Web scraping

AWS

Temporal

Async I/O

API integration

Pandas

Stripe

Ownership

Prompt engineering

Tools

Playwright

Selenium

Terraform

PostgreSQL

Job description

Sixtyfour is seeking an experienced Backend/Data Engineer to design agentic systems and scalable data pipelines in aSF-based environment. You will build search pipelines with OpenSearch, manage robust web scraping, and integrate third-party APIs for enrichment and search.

Strong Python+FastAPI, AWS, and prompt engineering experience are essential. Responsibilities include scaling workloads, handling CAPTCHA challenges, and ensuring fault-tolerant, cost-conscious operations across batch/streaming