Python Engineer – Web Crawling & Data Extraction

Certivity GmbH; United States; Hybrid; USD 101,000 - 157,000; Full time

Benefits offered by this job

EGYM Wellpass

Udemy access

30 vacation days per year (including 4

30 days workation in EU/selected coun.

Job summary

Certivity GmbH ищет опытного инженера по веб-краулингу и извлечению данных для разработки и эксплуатации масштабируемых систем. Вы будете отвечать за краулеры от обнаружения до защиты и вывода структурированных данных, используя Python и современные облачные технологии.

Вакансия предполагает работу в гибридном формате с вождением архитектурного направления и взаимодействием в небольшой команде. Требуется опыт работы с веб-скрапингом, Docker и облаками (GCP/AWS/Azure); знание LLM‑ассистированных

Qualifications

• 3+ года профессиональной разработки ПО (5+ на старшем треке).
• Сильный Python: продакшн код, асинхронность.
• Опыт веб-скрапинга на масштабах и работа с анти‑бот-защитами.
• Docker, опыт работы в одном крупном облаке (GCP/Azure/AWS)
• Git и CLI‑умения.
• Опыт с LLM‑assisted extraction или ML‑пайплайнами.
• Go или Java/Spring Boot.
• Frontend (React) или готовность освоить.
• Terraform, Kubernetes и современные DevOps‑инструменты.

Responsibilities

• Разрабатывать и масштабировать краулеры для больших сайтов и поддерживать их работу.
• Обход защит anti-bot: TLS/ fingerprinting, прокси, управление сессиями и лимитами.
• Преобразовать HTML/PDF в структурированные данные с точностью.
• Проектировать конвейеры ingestion, deduplication, версионирование и enrichment.
• Развертывать и поддерживать контейнеризованные приложения в облаке с CI/CD.
• Формировать архитектуру и культуру инженерии.
• Сотрудничать с командой и развивать стек технологий.

Skills

Python

Web scraping

LLM-assisted extraction

Async programming

Git

Tools

Docker

GCP

Azure

AWS

MongoDB

Kubernetes

React

Terraform

CI/CD

Job description

Описание: We build and operate large-scale web crawling and extraction systems that turn unstructured, dynamic websites into clean, structured and correct data. You'll own crawlers end-to-end: discovery, resilient fetching against real anti-bot defenses, and turning HTML/PDF into trustworthy structured output.

Задачи

Build and scale crawlers that handle dynamic, large-scale sites, and keep them running as those sites change and fight back. Stay ahead of anti-bot measures: TLS/browser fingerprinting, proxies, session and rate-limit strategy, and change detection. Turn raw HTML and PDF into clean, correct structured data (including LLM-assisted extraction), where silently wrong output is worse than a crash. Design and operate pipelines for ingestion, deduplication, versioning and enrichment. Ship and run containerized apps in the cloud, with CI/CD. Shape our technical architecture and engineering culture. Our Tech Stack Python, Docker, GCP (Cloud Run jobs), Azure Blob Storage, MongoDB. Go services and a React frontend on the periphery. You don't need all of it on day one, but you do need to ramp up fast.

Требования

3+ years of professional software development (5+ for the senior track). Strong Python: real production code, async. Hands-on web scraping at scale, including firsthand experience with anti-bot defenses and the failure modes of large-scale collection. A data-correctness mindset: you care that extracted data is right, not just that the job finished. Docker, at least one major cloud (GCP, Azure or AWS), Git and comfort on the CLI. A clear communicator who works well in a small, fast-moving team. LLM-assisted extraction or other work with LLMs, ML pipelines or data labeling workflows. Go or Java/Spring Boot. Frontend (React), or eagerness to pick it up. Terraform, Kubernetes and modern DevOps tooling.

Условия

A vibrant in-office culture in Munich, with the option to work from home up to 60% of the time. Flexible working hours. 30 vacation days per year (including 4 "company rest days" over Christmas). 30 days of "workation" per year, within the EU and selected countries. High autonomy and flat hierarchies. EGYM Wellpass for unlimited access to fitness courses and gyms. Udemy access for online courses. Closing Certivity is an equal opportunities employer. We are committed to equal employment opportunities regardless of race, religion, sexual orientation, age, marital status, disability, or gender identity. Please do not submit personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, data concerning your health, or data concerning your sexual orientation.