Lead Python Developer

VettedBench; India; On-site; INR 200,000 - 4,000,000; Full time

Benefits offered by this job

Flexible schedule

Health insurance

Internet reimbursement

Life insurance

Paid sick time

Job summary

VettedBench is seeking an experienced Python developer to build scalable web scraping systems. You will design and implement automated data extraction pipelines for diverse sources using Playwright, Selenium, and AWS services.

The role covers MFA and CAPTCHA flows, serverless architectures with Lambda and S3, and robust logging and testing. You will collaborate on architecture, drive best practices, and own code reviews and deployments.

Qualifications

• 10+ years of Python development with automation and data extraction.
• Proven web scraping expertise with Playwright/Selenium/Scrapy/BeautifulSoup/Requests.
• Experience with MFA, CAPTCHA solving, and session/cookie management.

Responsibilities

• Conceive and implement advanced scraping systems using Python and AWS.
• Automate access to JS-heavy, auth-secured sites including MFA/CAPTCHA.
• Design serverless scraping pipelines with Lambda, S3, IAM, CloudWatch.
• Scale data extraction with fault tolerance and robust logging.
• Coordinate across portals, APIs, and data sources.
• Contribute to architecture, tooling, and best practices.
• Lead code reviews and deployments.

Skills

Python

Automation

Data extraction

Web scraping

MFA handling

Cloud/AWS

Education

Bachelor’s or Master’s in CS/related

Tools

Playwright

Selenium

Scrapy

BeautifulSoup

Requests

Docker

Terraform

CloudFormation

Airflow

AWS Lambda

S3

IAM

CloudWatch

Secrets Manager

Job description

**Responsibilities**

• Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.
• Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session or token-based login flows.
• Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
• Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
• Integrate and manage complex workflows across multiple portals, APIs, and data sources.
• Contribute to architectural decisions, tooling, and best practices.
• Responsible for Code reviews and deployments.

**Qualifications**

• Bachelor’s or master’s degree in computer science, software engineering, or related field, or equivalent professional experience
• 10+ years of experience in Python development, with a strong focus on automation and data extraction
• Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests
• In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session or cookie management
• Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager
• Experience with asynchronous programming, headless browsers, and JavaScript-rendered content
• Solid understanding of web protocols (i.e. HTTP, HTTPS, cookies, headers), and the ability to reverse-engineer network calls and authentication mechanisms
• Comfortable working with APIs, JSON/XML, and data transformation
• Experience with CI or CD pipelines, Docker, and infrastructure as code (i.e. CloudFormation, Terraform)
• Familiar with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions
• Prior experience scraping from secure or enterprise-level portals (i.e. financial, healthcare, legal)
• Background in data engineering or ETL workflows is a plus
• Exposure to Python testing frameworks and writing unit or integration tests

We believe the best ideas and innovation happen when we are together. Employees in this role will work in the office four days, with the flexibility to work remotely one day (Friday).

Pay: ₹200,000.00 - ₹4,000,000.00 per year

Benefits:

• Flexible schedule
• Health insurance
• Internet reimbursement
• Life insurance
• Paid sick time

Work Location: In person