AI Scraping & LLM Engineering Manager – Python

AIMLEAP Inc; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

AIMLEAP Inc. is seeking an experienced engineer to architect and build AI-driven scraping and browser automation platforms.

You will lead the design of scalable pipelines for structured and unstructured web data, leveraging LLMs and modern AI models for automated extraction and enrichment. You will collaborate with AI, product, and engineering teams to deploy scalable scraping infrastructure using Scrapy, Selenium, Playwright, Docker, Kubernetes, and cloud-native systems.

Qualifications

• 9–15 years of experience in building large-scale web scraping, browser automation, and data extraction systems using Python.
• Proven expertise in Scrapy, Selenium, Playwright, Puppeteer, and modern browser automation frameworks for enterprise-grade scraping projects.
• Strong experience in building AI-powered scraping workflows using LLMs for intelligent extraction, data normalization, classification, and automation.
• Experience in designing scalable distributed scraping pipelines with proxy rotation, anti-bot handling, CAPTCHA solving, and high-concurrency crawling systems.
• Proven ability to lead engineering teams, manage delivery timelines, conduct code reviews, and drive architecture decisions for AI/data platforms.

Responsibilities

• Lead the architecture and development of AI-driven scraping and browser automation platforms for structured and unstructured web data extraction.
• Build intelligent scraping pipelines leveraging LLMs, LangChain, LlamaIndex, OpenAI, Claude, or open-source AI models for automated extraction and enrichment.
• Design and manage scalable scraping infrastructure using Scrapy, Selenium, Playwright, Docker, Kubernetes, queues, and cloud-native deployment systems.
• Collaborate with AI, product, and engineering teams to create next-generation intelligent data acquisition and automation solutions.
• Mentor and manage scraping/data engineering teams while establishing coding standards, best practices, monitoring, and scalable engineering processes.
• Optimize system performance, reliability, scalability, and deployment workflows using AWS/GCP, CI/CD pipelines, databases, and distributed processing systems.

Skills

Python

Scrapy

Selenium

Playwright

FastAPI/Django

APIs

LLMs

LangChain

Docker

Kubernetes

AWS/GCP

Education

Bachelor’s or Master’s degree in CS/Engineering/IT

Tools

Docker

Kubernetes

CI/CD

PostgreSQL

MongoDB

Redis

Job description

Position Details

Experience: 9 to 15 Years

Location: Remote (Work from Home) / Bangalore / India

Mode of Engagement: Full-time

No of Positions: 2

Educational Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT

Industry: AI / Data Engineering / Enterprise Automation / SaaS / LegalTech

Notice Period: Immediate / 15 Days Preferred

Job Description

What We Are Looking For

• 9–15 years of experience in building large-scale web scraping, browser automation, and data extraction systems using Python.
• Provenexpertisein Scrapy, Selenium, Playwright, Puppeteer, and modern browser automation frameworks for enterprise-grade scraping projects.
• Strong experience in building AI-powered scraping workflows using LLMs for intelligent extraction, data normalization, classification, and automation.
• Experience in designing scalable distributed scraping pipelines with proxy rotation, anti-bot handling, CAPTCHA solving, and high-concurrency crawling systems.
• Proven ability to lead engineering teams, manage delivery timelines, conduct code reviews, and drive architecture decisions for AI/data platforms.

Responsibilities

• Lead the architecture and development of AI-driven scraping and browser automation platforms for structured and unstructured web data extraction.
• Build intelligent scraping pipelines leveraging LLMs, LangChain, LlamaIndex, OpenAI, Claude, or open-source AI models for automated extraction and enrichment.
• Design and manage scalable scraping infrastructure using Scrapy, Selenium, Playwright, Docker, Kubernetes, queues, and cloud-native deployment systems.
• Collaborate with AI, product, and engineering teams to create next-generation intelligent data acquisition and automation solutions.
• Mentor and manage scraping/data engineering teams while establishing coding standards, best practices, monitoring, and scalable engineering processes.
• Optimizesystem performance, reliability, scalability, and deployment workflows using AWS/GCP, CI/CD pipelines, databases, and distributed processing systems.

Qualifications

• Bachelor’s orMaster’s degree in Computer Science, Engineering, IT, or related field.
• Strong hands-onexpertisein Python, Scrapy, Selenium, Playwright, FastAPI/Django, APIs, and scalable scraping architectures.
• Experience working with LLMs, AI automation frameworks, vector databases, semantic search, embeddings, and intelligent extraction systems.
• Proficiencyin Docker, Kubernetes, AWS/GCP, PostgreSQL/MySQL/MongoDB, Redis, CI/CD, and cloud infrastructure management.
• Prior experience leading engineering teams and delivering enterprise-scale scraping, AI automation, or data intelligence products.
• Excellent analytical, communication, leadership, and problem-solving skills with experience working in Agile development environments.