Data Platform Engineer III

The Standard India; Bengaluru; Hybrid; INR 4,500,000 - 6,500,000; Full time

Job summary

The Standard India is seeking a Data Platform Engineer III to lead pipeline and platform data engineering on our Data & AI Platform, including Databricks, Neo4j, and Confluent Kafka. You will own Lakebase ingestion, lineage, and freshness, with duties across the enterprise data foundation managed on Azure Databricks and related tech.

The role sits in a hybrid model with a Bengaluru base; on-site presence is essential for key engagements, with flexibility based on business needs.

Qualifications

• Typically requires 4+ years of data engineering experience building production pipelines.
• Strong PySpark and SQL skills and solid Python fundamentals.
• Hands-on Databricks experience: Delta Lake, jobs/workflows and Unity Catalog familiarity.
• Kafka/Confluent experience, including consuming event streams into analytical stores.
• Experience with pipeline orchestration (Lakeflow/Delta Live Tables, Airflow, Azure Data Factory or equivalent).
• Experience processing unstructured data (documents, text) in pipelines.
• Working knowledge of a graph or vector data store (Neo4j, Databricks Vector Search, Azure AI Search or equivalent).
• Data-quality testing experience (Great Expectations, dbt tests or equivalent) and lineage/freshness monitoring.
• Git-based workflows and CI/CD for data pipelines.

Responsibilities

• Build and operate batch and streaming ingestion pipelines on Databricks.
• Integrate event streams from Confluent Kafka into the Lakehouse and manage schema evolution.
• Build Lakebase-backed operational data patterns alongside the lakehouse.
• Develop ingestion and parsing pipelines for structured and unstructured content feeding the lakehouse, graph and vector stores.
• Operate and maintain Neo4j environments, including backups, monitoring and upgrades.
• Support index operations in Databricks Vector Search and Azure AI Search; monitor data freshness.
• Execute disaster recovery procedures for platform data stores with backup/restore testing and failover drills.
• Implement lineage capture and freshness monitoring against data contracts and SLAs.
• Automate data-quality and contract testing in CI/CD; respond to pipeline incidents.
• Maintain catalog and metadata hygiene in Unity Catalog; contribute runbooks and docs.

Skills

PySpark

SQL

Python

CI/CD

Git workflows

Data quality testing

Lineage & freshness monitoring

Observability tooling

On-call experience

Education

B.Tech / B.E. (Software/CS/IS)

Tools

Databricks

Delta Lake

Unity Catalog

Kafka / Confluent

Lakeflow

Airflow

Azure Data Factory

Neo4j

Databricks Vector Search

Azure AI Search

Job description

The next part of your journey is right around the corner — with Standard India

Data Platform Engineer III

Job Summary The Standard India is seeking a Data Platform Engineer III for pipeline and platform data engineering on our Data & AI Platform: Databricks, Lakebase, Confluent Kafka, ingestion pipelines, lineage, freshness and data movement. In this role, you will keep governed data products fresh, tested and within contract. You will also support day-to-day operations of the platform's graph and vector stores — Neo4j, Databricks Vector Search and Azure AI Search — under runbooks established by the Data Platform Engineer IV, including backup/restore and disaster recovery drills. Your initial assignment makes you the primary owner for Lakebase, ingestion, lineage and freshness handling on the Enterprise Memory task force, with supporting work on Databricks and data platform connectivity for the MCP Gateway task force. This role sits on The Standard's Data & AI Platform team, which owns the enterprise's shared data and AI foundation — governed once and consumed by many — built on Azure Databricks, Neo4j, Confluent Kafka, Kong, and Anthropic and OpenAI models behind a governed gateway.

Mode of Working

This role follows a hybrid work model, with a primary base in Bengaluru. While on-site presence is essential for key engagements, flexibility is offered for remote work based on business needs and team alignment. Ability to work on-site in Bengaluru, India is a requirement of the role.

Key Responsibilities

• Build and operate batch and streaming ingestion pipelines on Databricks (PySpark, Lakeflow/Delta Live Tables).
• Integrate event streams from Confluent Kafka into the Lakehouse and manage schema evolution.
• Build and operate Lakebase-backed operational data patterns alongside the lakehouse.
• Build ingestion and parsing pipelines for structured and unstructured content (documents, text) feeding the lakehouse, graph and vector stores.
• Operate and maintain Neo4j environments under established runbooks: backups and restores, monitoring, health checks and routine upgrades.
• Support vector index operations in Databricks Vector Search and Azure AI Search: run index refresh and sync pipelines and monitor retrieval data freshness.
• Execute disaster recovery procedures for platform data stores: backup/restore testing, failover drills and multi-region runbook execution.
• Implement lineage capture and freshness monitoring against published data contracts and SLAs.
• Automate data-quality and contract testing in CI/CD so violations fail fast, and respond to pipeline incidents.
• Maintain catalog and metadata hygiene in Unity Catalog, and contribute runbooks and operational documentation for the pipelines you own.

Skills And Background You’ll Need

Education: B.Tech / B.E. in Software Engineering, Computer Science or Information Systems is required.

Experience

• Typically requires 4+ years of data engineering experience building production pipelines.
• Strong PySpark and SQL skills and solid Python fundamentals.
• Hands-on Databricks experience: Delta Lake, jobs/workflows and familiarity with Unity Catalog.
• Kafka/Confluent experience, including consuming event streams into analytical stores.
• Experience with pipeline orchestration (Lakeflow/Delta Live Tables, Airflow, Azure Data Factory or equivalent).
• Experience processing unstructured data (documents, text) in pipelines.
• Working knowledge of a graph or vector data store (Neo4j, Databricks Vector Search, Azure AI Search or equivalent), including routine operations such as backups, monitoring and index refreshes.
• Data-quality testing experience (Great Expectations, dbt tests or equivalent) and lineage/freshness monitoring.
• Git-based workflows and CI/CD for data pipelines.

Preferred Experience

• Cypher basics and interest in growing into graph engineering.
• Disaster recovery or multi-region operations exposure (replication, failover drills, backup/restore testing).
• Lakebase or Postgres-style operational store experience.
• Confluent ecosystem depth (Schema Registry, connectors) and CDC tooling.
• Observability tooling (Datadog) and on-call experience.
• Good to have Databricks Data Engineer Associate or Confluent Certified Developer.

Key Behaviors Of a Successful Candidate

• Adaptability - Seeks information to understand the rationale for and importance of the change.
• Customer Focus - Displays an interest in the customer by trying to understand their concerns and issues; draws on customer insight to help others best meet current and future customer needs.
• Improvement Mindset - Demonstrates curiosity by asking questions regarding current approaches/methods and identifying potential changes.

Why Join Standard India?

• A rich benefits package that supports your health, well-being, and financial goals.
• An annual incentive bonus plan tied to both individual and organizational performance.
• Paid time off including earned leave, sick/casual leave, and public holidays, in accordance with the company's leave policy.
• A supportive, responsive management approach that empowers you to grow.
• Opportunities for career advancement through learning, coaching, and leadership enablement.
• A culture where your voice matters-your ideas shape our future, and we win as one.
• Purpose-driven work that makes a difference for our customers, communities, and each other.

In addition to a competitive salary, our employee-focused benefits are designed to support work-life balance, personal growth, and long-term impact.

Eligibility to participate in an incentive program is subject to the rules governing the program and plan. Any award depends on a variety of factors including individual and organizational performance.

StanCorp Global Services India Private Limited is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, religion, colour, sex, gender identity, sexual orientation, age, disability, caste, HIV status or any other characteristic protected by applicable laws of India.