Senior Web Scraping & Automation Engineer

Invisible Technologies; United States; Remote; USD 90,000 - 140,000; Full time

Job summary

Invisible Technologies in the United States is seeking a skilled Python crawler engineer to build and maintain advanced web scrapers.

You will leverage Scrapy, Selenium, BeautifulSoup and SQLAlchemy to collect data, manage pipelines, and overcome blocks, IP bans, and captchas.

The role involves deploying scripts to Linux-based AWS servers and collaborating with cross-functional teams in a fast-growing, automation-driven environment.

Qualifications

• Strong hands-on Python programming experience.
• Proficient with scraping libraries: Scrapy, Selenium, BeautifulSoup, Requests.
• 3+ years with web frameworks: Flask, Django, FastAPI, Tornado.
• Strong SQL knowledge with relational databases.
• Git version control and REST API basics.
• Familiarity with JavaScript, HTML5 and CSS3.

Responsibilities

• Create crawlers for source URLs using Python modules including Scrapy, Selenium, Requests, BeautifulSoup, and Splash
• Maintain Scrapy pipelines and middlewares to manage crawler output
• Develop crawlers capable of overcoming technical roadblocks like IP bans, geolocation bans, and captchas
• Write SQL queries to manage database operations using SQLAlchemy
• Deploy Python scripts and crawlers to Linux-based AWS servers

Skills

Python

REST APIs

JavaScript

HTML5

CSS3

OOP

Git

Tools

Scrapy

Selenium

BeautifulSoup

Requests

SQLAlchemy

SQL

Job description

Invisible Technologies in the United States is seeking a skilled Python crawler engineer to build and maintain advanced web scrapers.

You will leverage Scrapy, Selenium, BeautifulSoup and SQLAlchemy to collect data, manage pipelines, and overcome blocks, IP bans, and captchas.

The role involves deploying scripts to Linux-based AWS servers and collaborating with cross-functional teams in a fast-growing, automation-driven environment.