Lead Data Platform Engineer OpenSearch

Arrow Electronics; Vijaynagar; On-site; INR 2,400,000 - 5,000,000; Full time

Benefits offered by this job

Competitive Salary

Work Life Balance

Professional Development

Innovative Work Environment

Job summary

Arrow Electronics is seeking a Lead Data Platform Engineer (OpenSearch) to architect and optimize large-scale data pipelines in AWS, Kafka, and OpenSearch. The role requires strong data modeling, SQL tuning, and deployment governance across teams.

You will design high-availability data pipelines, optimize storage and indexing, and guide backend teams on best practices for data infrastructure. Eligible candidates have extensive AWS, Kafka, and OpenSearch experience.

Qualifications

• Over 7 years of experience in data engineering or related backend development.
• Extensive experience in operating infrastructure handling large-scale traffic in AWS cloud environments.
• Strong capabilities in building real-time data pipelines and troubleshooting using Kafka.
• Experience in OpenSearch (Elasticsearch) cluster design and sharding optimization.
• Deep understanding of the internal structure of RDBMS (PostgreSQL/MySQL) and ability to tune complex queries.

Responsibilities

• High-availability data pipeline design: AWS MSK (Kafka), end-to-end data streaming architecture design and operation using Logstash.
• Storage optimization and sharding strategies: Optimizing large-scale search performance through OpenSearch's index design, sharding, and lifecycle (ISM) management.
• Data Modeling and Partitioning: Table partitioning and data modeling in PostgreSQL/MySQL, architecture guide for efficient querying.
• Performance Tuning and Guidance: Leading SQL optimization across services and spreading technical best practices to backend development teams.
• Release Management: Be the gatekeeper during key database / data pipeline release activities, create review checklist, review deployment details and validates details before deployment to ensure zero-error deployments without roll back needs.
• Infrastructure automation: DB schema versioning (such as Flyway) and deployment process automation systems.

Skills

Data engineering

AWS cloud

Kafka

OpenSearch

PostgreSQL/MySQL

Education

Bachelor’s degree in CS or equivalent

Tools

Kafka

OpenSearch

PostgreSQL/MySQL

Flyway

Terraform

Ansible

Logstash

AWS

Job description

Job Description:

Lead Data Platform Engineer (OpenSearch)

Role: Lead Data Platform Engineer (AWS | Kafka | OpenSearch)

Preferred Location: Ahmedabad, Pune, Indore

Years of Experience: 08 – 12 years

Notice Period – 0-30 Days (Preferred)
Introduction
Our team builds a data platform that reliably collects, processes, and retrieves large-scale IoT data based on AWS. We are seeking senior engineer who can design unobstructed flow even among billions of data records, push data storage performance to the limit, and technically lead service growth.

Key Responsibilities

• High-availability data pipeline design: AWS MSK (Kafka), end-to-end data streaming architecture design and operation using Logstash
• Storage optimization and sharding strategies: Optimizing large-scale search performance through OpenSearch's index design, sharding, and lifecycle (ISM) management
• Data Modeling and Partitioning: Table partitioning and data modeling in PostgreSQL/MySQL, architecture guide for efficient querying
• Performance Tuning and Guidance: Leading SQL optimization across services and spreading technical best practices to backend development teams
• Release Management: Be the gatekeeper during key database / data pipeline release activities, create review checklist, review deployment details and validates details before deployment to ensure zero-error deployments without roll back needs.
• Infrastructure automation: DB schema versioning (such as Flyway) and deployment process automation systems

Requirements

• Over 7 years of experience in data engineering or related backend development
• Extensive experience in operating infrastructure handling large-scale traffic in AWS cloud environments
• Strong capabilities in building real-time data pipelines and troubleshooting using Kafka
• Experience in OpenSearch (Elasticsearch) cluster design and sharding optimization is essential.
• Deep understanding of the internal structure of RDBMS (PostgreSQL/MySQL) and ability to tune complex queries
  

Preferred Qualifications

• Seamless technical collaboration capabilities: Communication skills to discuss infrastructure optimization with DevOps engineers and provide efficient data modeling and query guides to backend developers
• Global collaboration skills: Able to share technical documents with engineers across the globe and conduct work via video conferences during collaborative work sessions
• IoT Domain Experience: Experience in handling large-scale device messages and building data pipelines
• Ensuring data stability: Experience in designing logic to prevent data loss during failures and ensure consistency verification
• Nice to have è IaC Proficiency: Nice to have Experience coding and automated operations of data infrastructure using Terraform, Ansible, and others
• 4 years of bachelor’s degree or higher in Computer Science or equivalent discipline

What We Offer:

• Competitive Salary
• Work Life Balance
• Professional Development
• Innovative Work Environment.
• Recognition and Rewards

Location: IN-GJ-Ahmedabad, India-Ognaj (eInfochips)

Time Type: Full time

Job Category: Engineering Services

Requirements: