Senior Engineering Lead – Data Connectors & Ingestion Platform

NorthStar HR Consultants; Maharashtra; On-site; INR 800,000 - 1,400,000; Full time

Job summary

NorthStar HR Consultants is seeking a Senior Technical Lead for Data Connectors & Ingestion Platform in Pune. Lead backend teams responsible for data collection, ingestion, and analytics, driving architecture and delivery across hundreds of third-party connectors.

You will own roadmap, collaborate with Product, Security, Solutions Engineering, and Customer Success, and ensure scalable, reliable, and high-performance integrations for diverse SaaS, IAM, and cloud sources.

Qualifications

• 15+ years of software engineering experience building distributed backend systems.
• 3+ years leading engineering teams or acting as a technical lead.
• Strong experience building API integrations with third-party platforms.
• Experience owning engineering roadmaps and delivery.

Responsibilities

• Lead and grow a team of backend engineers responsible for data collection, ingestion and analytics platform.
• Own engineering execution, planning, estimation, delivery, and operational excellence.
• Drive technical decisions and architectural direction for the integrations platform.
• Mentor engineers and establish a strong engineering culture focused on quality, ownership, and continuous improvement.
• Hire, coach, and develop high-performing engineers.
• Own the delivery roadmap for new connectors, enhancements, and platform capabilities.
• Work closely with Product Management to prioritize integrations based on customer needs and business priorities.
• Balance feature development with technical debt reduction and platform scalability.
• Ensure predictable delivery through effective sprint planning and execution.
• Design scalable frameworks for building and maintaining hundreds of third-party connectors.
• Improve connector architecture to maximize code reuse, consistency, and maintainability.
• Drive standardization of authentication mechanisms, API clients, data ingestion pipelines, retries, rate limiting, observability, and error handling.
• Ensure the platform supports REST APIs, GraphQL, webhooks, event streaming, log collection, SDKs, and file-based ingestion.
• Establish engineering best practices around: Automated testing, CI/CD, Reliability, Performance, Monitoring, Documentation.
• Improve deployment automation and release confidence.
• Drive operational metrics such as availability, MTTR, connector reliability, and engineering productivity.

Job description

Job Title - Senior Engineering / Technical Lead – Data Connectors & Ingestion Platform (70% technical & 30% people mgmt)

Job Location - Pune, Maharashtra

MUST Have Skills - Golang or Python, data pipeline , data connectors / collectors experience, Exp with SaaS products, Strong experience building API integrations

Product

The product is heavily driven by data analytics as well as AI/ML. The core idea is to analyze events & logs from enterprise systems & networks to enable the use cases we are trying to solve in the area of identity & access security.

Position Overview

As customers adopt hundreds of SaaS applications, cloud platforms, IAM providers, security products, and infrastructure technologies, building reliable integrations becomes one of the most critical engineering investments.

Our client is looking for an experienced Senior Technical Lead to own this platform.

You will own the engineering team responsible for building, maintaining, and evolving ecosystem of third-party integrations.

Your team develops scalable, reliable, and high-performance connectors that ingest security and identity data from cloud providers, IAM systems, SaaS applications, endpoint platforms, SIEMs, networking products, and enterprise infrastructure.

You will own engineering execution—from roadmap planning through production operations—while partnering closely with Product Management, Architecture, Security Data Engineering, Solutions Engineering, and Customer Success.

This role combines technical leadership, software architecture, execution excellence, and people leadership.

Responsibilities

Engineering Leadership

• Lead and grow a team of backend engineers responsible for data collection, ingestion and analytics platform.
• Own engineering execution, planning, estimation, delivery, and operational excellence.
• Drive technical decisions and architectural direction for the integrations platform.
• Mentor engineers and establish a strong engineering culture focused on quality, ownership, and continuous improvement.
• Hire, coach, and develop high-performing engineers.

Product & Delivery Ownership

• Own the delivery roadmap for new connectors, enhancements, and platform capabilities.
• Work closely with Product Management to prioritize integrations based on customer needs and business priorities.
• Balance feature development with technical debt reduction and platform scalability.
• Ensure predictable delivery through effective sprint planning and execution.
• Design scalable frameworks for building and maintaining hundreds of third-party connectors.
• Improve connector architecture to maximize code reuse, consistency, and maintainability.
• Drive standardization of authentication mechanisms, API clients, data ingestion pipelines, retries, rate limiting, observability, and error handling.
• Ensure the platform supports REST APIs, GraphQL, webhooks, event streaming, log collection, SDKs, and file-based ingestion.
• Establish engineering best practices around:
• Automated testing
• CI/CD
• Reliability
• Performance
• Monitoring
• Documentation
• Improve deployment automation and release confidence.
• Drive operational metrics such as availability, MTTR, connector reliability, and engineering productivity.

Cross-functional Collaboration

Partner closely with:

• Product Management
• Security Data Engineering
• Solutions Engineering
• Customer Success
• Translate customer onboarding requirements into scalable engineering solutions.

Operational Ownership

• Own production health of all integration services.
• Lead incident management and postmortems.
• Improve system observability and monitoring.
• Ensure integrations continue to perform as third-party APIs evolve.
• Drive proactive improvements to reliability and scalability.

Continuously evaluate new technologies, APIs, and integration patterns to improve developer productivity and platform capabilities.

Drive automation / AI based development wherever possible to reduce the engineering effort required to build and maintain integrations.

Requirements

Experience

• 15+ years of software engineering experience building distributed backend systems.
• 3+ years leading engineering teams or acting as a technical lead.
• Strong experience building API integrations with third-party platforms.
• Experience owning engineering roadmaps and delivery.

Technical Skills

Strong experience with:

• Distributed systems
• Backend services
• REST APIs
• GraphQL
• Webhooks
• Event-driven architectures
• Microservices
• Asynchronous processing
• Cloud-native applications

Preferred technologies:

• Go (Preferred), Java, or Python
• Docker
• Kafka or similar messaging systems
• PostgreSQL, Click House, Elasticsearch, or similar databases

Leadership Skills

• Strong engineering leadership and execution skills.
• Excellent project planning and prioritization.
• Experience mentoring senior engineers.
• Strong communication and stakeholder management.
• Ability to balance speed with engineering quality.
• Comfortable working in a fast-paced startup environment.

Nice to Have

• Experience building connector platforms or integration frameworks.
• Familiarity with IAM, identity security, cybersecurity, or SIEM technologies.
• Experience working with APIs from cloud providers such as Microsoft, Google, AWS, Okta, Duo, CrowdStrike, Zscaler, Palo Alto, or Microsoft Defender.
• Experience building multi-tenant SaaS platforms.
• Exposure to large-scale data processing and streaming systems.