Senior Data Engineer

Straive; Hyderabad, Gurugram District, Chennai; Híbrido; INR 1.800.000 - 2.600.000; Jornada completa

Descripción de la vacante

Straive is seeking a Senior Data Engineer to architect and build foundational data infrastructure powering enterprise-grade Generative AI and autonomous agent systems. You will design real-time, federated data substrates for AI agents to access, reason over, and orchestrate data in milliseconds.

Responsibilities include building agent-ready pipelines, federated architectures, optimized data models, identity resolution, data governance, and cost-aware pipeline orchestration using Airflow and

Formación

• 6+ years of hands-on data engineering experience in enterprise environments.
• Strong experience with Google Cloud Platform data services (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
• Proven ability to design streaming architectures and event-driven data feeds beyond batch processing.
• Expert SQL optimization and schema design for OLAP/OLTP patterns.
• Hands-on experience with Apache Airflow or Cloud Composer for DAG orchestration.
• Experience with distributed processing engines like Spark or Hadoop.

Responsabilidades

• Design and deploy low-latency, real-time data pipelines feeding AI agents.
• Build read-only, federated data layers and API wrappers for cross-system querying.
• Model data schemas optimized for LLM tool-calling and agent context.
• Implement data governance, DLP, and ABAC for regulatory compliance.
• Orchestrate pipelines using Airflow/Cloud Composer to balance cost and performance.
• Collaborate with AI Architects to align data assets with agent frameworks.

Conocimientos

Data engineering

GCP mastery

Real-time systems

Advanced SQL

Data modeling

Airflow/Cloud Composer

Distributed processing

Herramientas

BigQuery

Dataflow

Pub/Sub

Cloud Composer

Spark/Hadoop

Descripción del empleo

Job Description: Senior Data Engineer

Job Title: Senior Data Engineer (Agentic Data Infrastructure)

Experience Range: 6+ Years (with strong focus on modern cloud data architectures)

Location: Bangalore / Hyderabad / Gurgaon/Chennai (Hybrid)

Job Overview
We are seeking a highly skilled Senior Data Engineer to architect and build the foundational data infrastructure that powers enterprise-grade Generative AI and autonomous agent systems. Rather than simply building traditional batch-oriented data warehouses, you will be designing real-time, federated data substrates that allow AI agents to securely access, reason over, and orchestrate complex enterprise data in milliseconds.
In this role, you will solve systemic data fragmentation problems by building zero-copy architectures, low-latency event streams, and dynamic access layers across heterogeneous enterprise platforms. You will work closely with AI Architects to ensure data structures (whether relational, graph, or vector) are optimized specifically for LLM tool-calling and agentic reasoning under strict regulatory constraints.

Key Responsibilities

• Agent-Ready Data Pipelines: Design and deploy low-latency, event-driven data pipelines (GCP Pub/Sub, Dataflow) that feed real-time context to stateless AI agents, shifting away from traditional nightly batch ETLs.


• Federated Data Architecture: Build read-only integration layers and API wrappers that allow AI models to query disparate operational systems (legacy ledgers, risk engines, CRMs) in parallel without massive data duplication.


• Optimized Data Modeling for AI: Design sophisticated BigQuery schemas, Spanner/AlloyDB relational models, and potentially Graph or Vector structures specifically tuned for high-concurrency, low-latency LLM tool-calling and context injection.


• Identity Resolution & Data Joining: Solve complex cross-system identity mapping challenges, ensuring AI agents have a unified, accurate "360-degree" view of entities across fragmented enterprise brands and databases.


• Governance & Pipeline Security: Implement automated data quality checks, PII redaction layers (Cloud DLP), and Attribute-Based Access Controls (ABAC) to ensure agents never ingest or expose restricted regulatory data.


• Pipeline Orchestration & Cost Tuning: Orchestrate complex DAGs using Apache Airflow (Cloud Composer) while aggressively monitoring and optimizing compute resources (minimizing shuffle in Dataflow/Spark) to control cloud costs at enterprise scale.

Required Qualifications

• Data Engineering Experience: 6+ years of hands-on experience building robust, scalable data pipelines and modern data warehouses in enterprise environments.


• Google Cloud Platform (GCP) Mastery: Strong operational experience with GCP data services, specifically BigQuery, Dataflow (Apache Beam), Pub/Sub, and Cloud Storage.


• Real-Time & Event-Driven Systems: Proven ability to design streaming architectures and asynchronous data feeds, moving beyond traditional batch processing.


• Advanced SQL & Data Modeling: Expert-level SQL optimization (joins, window functions) and experience designing schemas for both analytical (OLAP) and transactional/operational (OLTP) access patterns.


• Pipeline Orchestration: Hands-on experience with Apache Airflow (Cloud Composer) or similar modern orchestration tools.


• Distributed Frameworks: Working knowledge of distributed processing engines like Spark or Hadoop for heavy lifting when required.


• Cross-Functional Problem Solving: Ability to tackle ambiguous data problems (e.g., identity resolution, federated querying) across siloed enterprise teams.

What Makes You Stand Out

• Experience building data layers specifically designed for consumption by LLMs or autonomous agents (e.g., semantic layers, feature stores for AI).


• Familiarity with low-latency operational databases on GCP (Cloud Spanner, AlloyDB, or Memorystore/Redis).


• Experience in heavily regulated industries (FinTech, Banking, Healthcare) handling sensitive PII/financial data with strict compliance mandates.


• Understanding of API integration and how to expose data assets as tools for Model Context Protocol (MCP) or similar agent frameworks.