Remote Senior Backend Engineer – Scalable Web Crawling

CyberCoders; Redwood City (CA); On-site; USD 150,000 - 185,000; Full time

Job summary

CyberCoders is seeking a senior Back-End Software Engineer to design, implement, and operate scalable web crawling, data ingestion, and processing systems. You will build distributed services, use headless browsers, and optimize APIs for downstream teams.

The role emphasizes anti-bot strategies, proxy management, and secure data handling, with containerized deployments on cloud platforms and strong focus on reliability and performance at scale.

Qualifications

• 5+ years building back-end systems with production experience in Python, Go, Java, Node.js or Rust.
• Experience with distributed systems, microservices, concurrency, and message brokers.
• Proven experience with web crawling, scraping stacks, headless browsers, HTTP clients, and HTML parsing.
• Familiarity with proxy management, CAPTCHA mitigation, fingerprinting, and anti-bot strategies.
• Knowledge of databases, search/indexing, and data pipelines.

Responsibilities

• Design, implement, and maintain scalable microservices for crawling, scraping, and data ingestion.
• Develop headless browser pipelines and robust HTML parsing with modern tooling.
• Build and operate REST and GraphQL APIs for downstream use.
• Implement concurrency and queues to maximize throughput with controlled resource usage.
• Architect data pipelines with Kafka, RabbitMQ, AWS SQS, and ETL schedulers like Airflow.
• Work with Docker/Kubernetes and cloud platforms (AWS, GCP, Azure) for scalable deployments.
• Instrument systems with observability stacks and CI/CD pipelines.
• Mentor engineers and contribute to architecture and code reviews.

Skills

Python

Go

Java

Node.js

Rust

Distributed systems

Kafka

RabbitMQ

AWS SQS

Selenium

Puppeteer

Playwright

BeautifulSoup

lxml

Cheerio

Docker

Kubernetes

AWS

GCP

Azure

GitHub Actions

Jenkins

Git

Airflow

Celery

Requests

AIOHTTP

PostgreSQL

MySQL

MongoDB

Redis

Elasticsearch

Education

Bachelor/Master in CS/Engineering

Tools

Puppeteer

Playwright

Selenium

BeautifulSoup

lxml

Cheerio

Docker

Kubernetes

AWS

GCP

Azure

ELK

Prometheus

Grafana

Datadog

GitHub Actions

Jenkins

Git

Airflow

Celery

Requests

AIOHTTP

PostgreSQL

MySQL

MongoDB

Redis

Elasticsearch

Job description

CyberCoders is seeking a senior Back-End Software Engineer to design, implement, and operate scalable web crawling, data ingestion, and processing systems. You will build distributed services, use headless browsers, and optimize APIs for downstream teams.

The role emphasizes anti-bot strategies, proxy management, and secure data handling, with containerized deployments on cloud platforms and strong focus on reliability and performance at scale.