Data Pipelines Engineering Manager – Python

FullStackTechies; Bengaluru; On-site; INR 2,000,000 - 3,000,000; Full time

Job summary

FullStackTechies is seeking a Data Pipelines Engineering Manager to lead engineering teams in designing and implementing robust data solutions. The ideal candidate must have 7-12 years of experience in data management and extensive knowledge of Python, AWS, and API architectures. Responsibilities include overseeing data ingestion processes, mentoring engineers, and ensuring high-quality data governance. This role offers the flexibility of remote work or can be based in Bangalore, India.

Qualifications

• 7-12 years of experience in managing end-to-end data systems.
• Proven experience using large language models to automate and scale web data extraction.
• Strong background in building scalable APIs and microservices using Python.

Responsibilities

• Lead and mentor engineering teams responsible for data ingestion and transformation.
• Architect data systems that support analytics and product intelligence.
• Oversee API architecture and system scalability.

Skills

Python

Django

FastAPI

SQL

Data Pipelines

Scraping Frameworks

AI/LLM integration

Docker

Kubernetes

Git-based CI/CD workflows

Education

Bachelor’s or master’s degree in computer science, engineering, or related field

Tools

AWS cloud services

PostgreSQL

MySQL

Job description

Data Pipelines Engineering Manager – Python

Experience: 7-12 Years Location: Remote (Work from Home) / Bangalore / India Mode of Engagement: Full-time No of Positions: 2 Educational Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / LegalTech / Enterprise Solutions Notice Period: Immediate

Job Description:

What We Are Looking For

• 7-12 years of experience in managing end-to-end data systems, including data collection, API-driven data delivery, and AI model integrations.
• Proven experience using large language models to automate and scale web data extraction, including building custom AI scraping pipelines for structured data collection at scale
• Proven expertise in Python-based data engineering, data scraping, and pipeline orchestration for large-scale analytics and automation projects.
• Strong background in building scalable APIs, microservices, and data pipelines using Python, Docker, and Kubernetes.
• Experience in customizing open-source language models (LLMs) and integrating them into enterprise solutions.
• Proficiency in AWS cloud services, PostgreSQL/MySQL, CI/CD, and DevOps tools for continuous delivery and monitoring.
• Ability to lead distributed engineering teams, manage delivery timelines, and drive innovation through automation and AI-driven workflows.

Responsibilities:

• Lead and mentor engineering teams responsible for designing, implementing, and maintaining data ingestion, transformation, and delivery pipelines.
• Architect data systems combining scraping, big data processing, and AI-based enrichment to support analytics and product intelligence.
• Oversee API architecture and system scalability, ensuring high performance and secure data flow across services.
• Manage public data scraping and data integration projects that involve large, unstructured datasets and complex relational mappings.
• Drive AI model customization using open-source frameworks (LangChain, LlamaIndex, or custom LLM integrations) to improve automation and search intelligence.
• Collaborate with product managers, AI teams, and data analysts to translate requirements into actionable data engineering tasks.
• Implement and monitor data quality, governance, and compliance standards throughout the data lifecycle.
• Optimize deployment processes using Docker, Kubernetes, AWS Lambda, and CI/CD pipelines.
• Conduct regular technical reviews, mentor developers, and establish best practices for Python, cloud deployment, and automation.

Qualifications:

• Bachelor’s or master’s degree in computer science, engineering, or related field.
• Hands‑on expertise in Python, Django, FastAPI, SQL, and API-driven system design.
• Strong understanding of data pipelines, scraping frameworks (BeautifulSoup, Scrapy), and distributed data systems.
• Experience with AI/LLM integration, automation frameworks, and data-driven architecture design.
• Proficiency in AWS cloud ecosystem, Docker, Kubernetes, and Git-based CI/CD workflows.
• Demonstrated leadership in managing multi‑member engineering teams and delivering enterprise‑grade products.
• Excellent analytical, communication, and problem‑solving abilities.
• Familiarity with Agile methodologies, Jira/ClickUp, and project management best practices.