Senior Automation Engineer — Scalable Web Scraping

KWAN; Porto; On-site; EUR 55,000 - 75,000; Full time

Job summary

KWAN in Porto seeks a Senior Automation Engineer to design and scale data automation and web scraping pipelines using Python, with a strong focus on reliability, performance, and maintainability.

You will integrate diverse data sources, including sites with anti-bot measures, build REST API integrations and event-driven workflows, apply AI-assisted extraction, monitor data quality, and collaborate with cross-functional teams to deliver robust data products.

Qualifications

• 5+ years in automation and data engineering, Python-first.
• Industrial-scale web scraping: Playwright or Selenium, session management, anti-bot, captchas, proxies, rate limiting.
• Speed in integrating new sources — marginal cost of adding a source defines scaling.
• API integration where available: REST, OAuth, webhooks, pagination.
• Extraction of unstructured information: PDFs, news, documents — analysis, entity extraction, normalization, deduplication.
• Public sources of statistical data (Eurostat, INE, World Bank and equivalents).
• Orchestration and reliability: scheduling, retries, idempotence, monitoring, alerting.
• Source quality: validating collections and detecting incomplete data.
• Git, code review, CI/CD and automated testing.
• Fluent level of English and Portuguese.

Responsibilities

• Design and build Python-based automation and web scraping solutions at industrial scale.
• Integrate new data sources quickly, including sites with anti-bot measures, captchas, proxies and rate limiting.
• Build and maintain integrations via REST APIs, OAuth and webhooks, including paginated data sources.
• Extract and normalize information from unstructured sources such as PDFs, reports and news articles.
• Implement orchestration, scheduling, retries, idempotency, monitoring and alerting for data pipelines.
• Validate data quality at the source and detect silent or incomplete collection failures.
• Apply LLMs and agent-based approaches for document/page extraction, news classification and summarization, and self-healing scrapers.

Skills

Python

Web scraping

Data engineering

REST APIs

CI/CD

English

Portuguese

Tools

Playwright

Selenium

Git

CI/CD

Job description

KWAN in Porto seeks a Senior Automation Engineer to design and scale data automation and web scraping pipelines using Python, with a strong focus on reliability, performance, and maintainability.

You will integrate diverse data sources, including sites with anti-bot measures, build REST API integrations and event-driven workflows, apply AI-assisted extraction, monitor data quality, and collaborate with cross-functional teams to deliver robust data products.