Data Engineering Manager – Web Crawling & Pipeline Architecture

Outsource Bigdata Solutions; Bengaluru; Remote; INR 1,800,000 - 2,500,000; Full time

Benefits offered by this job

Flexible working hours

Health benefits

Career development opportunities

Job summary

A leading IT solutions company based in India is seeking a Data Engineering Manager specializing in web crawling and pipeline architecture. The ideal candidate should have 7–12 years of experience, and strong expertise in Python and SQL. This role involves leading a team, optimizing data pipelines, and integrating AI-driven automation into workflows. The position is full-time and remote, offering exposure to innovative technologies in data management and engineering.

Qualifications

• 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems.
• Strong expertise in Python and SQL.
• Hands-on experience with cloud data platforms (AWS/GCP/Azure).

Responsibilities

• Lead and mentor data engineering and web crawling teams, ensuring high-quality delivery.
• Architect, implement, and optimize scalable data pipelines.
• Integrate AI agents or LLM-based automation for data tasks.

Skills

Python

SQL

Data pipeline design

Web crawling

AI-driven automation

Cloud platforms (AWS/GCP/Azure)

Workflow orchestration tools (Airflow/Celery)

Education

Bachelor’s or master’s degree in engineering or Computer Science

Tools

Airflow

Celery

RPA tools

Job description

Data Engineering Manager – Web Crawling & Pipeline Architecture

Experience: 7 to 12 Years
Location: Remote / Bangalore
Engagement: Full-time
Positions: 2
Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry: IT / Data / AI / E-commerce / FinTech / Healthcare
Notice Period: Immediate

What We Are Looking For:

• Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
• Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
• Hands‑on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
• Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
• Deep understanding of web crawling frameworks, proxy rotation, anti‑bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA‑safe crawling).
• Strong expertise in AI‑driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..

Responsibilities:

• Lead and mentor data engineering and web crawling teams, ensuring high‑quality delivery and adherence to best practices.
• Architect, implement, and optimize scalable data pipelines that support high‑volume data ingestion, transformation, and storage.
• Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.
• Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.
• Define and enforce data quality, validation, and security measures across all data flows and pipelines.
• Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions.
• Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.
• Oversee cloud‑based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure.
• Integrate AI agents or LLM‑based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling.

Qualifications:

• Bachelor’s or master’s degree in engineering, Computer Science, or related field.
• 7–12 years of relevant experience in data engineering, pipeline design, or large‑scale web crawling systems.
• Strong expertise in Python, SQL, and modern data processing practices.
• Experience working with Airflow, Celery, or similar workflow automation tools.
• Solid understanding of proxy systems, anti‑bot techniques, and scalable crawler architecture.
• Hands‑on experience with cloud data platforms (AWS/GCP/Azure).
• Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).
• Strong analytical, architectural, and leadership skills.

About Us:

AIMLEAP is an ISO 9001:2015 and ISO/IEC 27001:2013 certified global technology consulting and service provider offering Digital IT, AI‑augmented Data Solutions, Automation, and Research & Analytics Services.

AIMLEAP has been recognized as ‘The Great Place to Work®’. With focus on AI and automation‑first approach, our services include end‑to‑end IT application management, Mobile App Development, Data Management, Data Mining Services, Web Data Scraping, Self‑serving BI reporting solutions, Digital Marketing, and Analytics solutions.

We started in 2012 and successfully delivered projects in IT & digital transformation, automation driven data solutions, and digital marketing for more than 750 fast‑growing companies in the USA, Europe, New Zealand, Australia, Canada; and more.

– An ISO 9001:2015 and ISO/IEC 27001:2013 certified
– Served 750+ customers
– 12+ Years of industry experience
– 98% Client Retention
– Great Place to Work® Certified
– Global Delivery Centers in the USA, Canada, India & Australia.