Senior Data Engineer

Straive; Bengaluru; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Straive is seeking a Senior Data Engineer to design, build, and optimize an enterprise data platform and analytics solutions. You will transform complex business requirements into scalable, production-grade data pipelines and Lakehouse architectures using Azure Databricks, Microsoft Fabric, Python, PySpark, and SQL.

You will work on API/database integrations, data quality, and AI/ML readiness, preparing data pipelines for downstream analytics, LLMs, and intelligent applications in Bengaluru with

Qualifications

• Core development in Python, PySpark, and SQL for large-scale data transformations.

Responsibilities

• Design, implement, and maintain high-performance batch and real-time data pipelines.
• Build Medallion Lakehouse architectures (Bronze, Silver, Gold) using Delta Lake principles.
• Develop API connectors and data integration pipelines from heterogeneous sources.
• Prepare data pipelines for AI/ML workloads and ensure readiness for downstream analytics and LLMs.

Skills

Python

PySpark

SQL

Azure Databricks

Microsoft Fabric

Delta Lake

Genie

Unity Catalog

Education

Bachelor’s or Master’s degree in Computer Science/IT

Tools

Azure Databricks

Genie

Microsoft Fabric

Power BI

Delta Lake

Job description

Location: Straive Locations

Experience Level: 5–8 Years

Job Summary

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize our enterprise

data platform and analytics solutions. In this role, you will serve as a key hands-on engineering track

lead responsible for transforming complex business requirements into scalable, production-grade data

pipelines and Lakehouse architectures.

You will work heavily across Azure, Databricks, Microsoft Fabric, Python, PySpark, and SQL to

execute robust Medallion Lakehouse implementations (Bronze, Silver, Gold), build resilient

API/database integrations, and engineer clean data structures. Additionally, you will play an active

role in preparing data pipelines for modern AI/ML workloads, ensuring our data platform is fully

optimized for downstream analytics, LLMs, and intelligent applications.

Key Responsibilities

Data Pipeline & Lakehouse Engineering: Design, implement, and maintain high-performance

batch and real-time streaming data pipelines using Python, PySpark, and Spark SQL on

Azure Databricks and Microsoft Fabric. Build multi-tier Medallion architectures (Bronze, Silver,

Gold) using Delta Lake principles. Good experience on Genie.

Microsoft Fabric Platform Delivery: Leverage Fabric capabilities (OneLake, Direct Lake,

Lakehouses, and Data Factory Gen2) to streamline zero-copy ingestion, optimize semantic

models, and eliminate redundant ETL pipelines.

API & Systems Integration: Build resilient API connectors, CDC workflows, and data

integration pipelines to ingest data from heterogeneous sources (relational databases, flat

files, third-party REST APIs, and enterprise cloud applications).

AI & Analytics Enabling: Structure, clean, and optimize lakehouse data layers for downstream

Power BI reporting as well as AI/ML applications, ensuring data is clean, validated, and

structured for AI grounding.

Data Quality & Governance: Implement robust data validation, access controls, data lineage,

and metadata management using Unity Catalog, Fabric Governance, and automated testing

frameworks.

Technical Mentorship & CI/CD: Perform thorough code reviews, enforce engineering best

practices, optimize query performance, and manage automated deployment pipelines using

Azure DevOps or GitHub Actions.

Technical Skills &amp Qualifications

Primary Requirements (Must-Have)

• 1. Core Development Languages: Advanced hands-on proficiency in Python, PySpark, and SQL (Spark SQL / T-SQL) for large-scale data manipulation, performance tuning, and complex transformations.
• 2. Databricks Platform: Strong hands-on experience with Azure Databricks (Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader) and Genie.
• 3. Microsoft Fabric: Demonstrated experience building pipelines and workspace items within Microsoft Fabric (OneLake, Fabric Lakehouses/Warehouses, Data Factory Gen2, Direct Lake semantic models).
• 5. AI Data Concepts & AI Readiness: Strong conceptual and practical understanding of data (Retrieval-Augmented Generation) data ingestion, feature stores, and structuring unstructured/semi-structured data for LLMs. handling schema drift, parsing complex JSON/REST API payloads, and modeling Star Schema / Dimensional Gold layers.

Preferred Skills (Good-to-Have)

• Agentic AI Concepts & Frameworks: Familiarity with Agentic AI architecture patterns (e.g., LangGraph, AutoGen, or vector stores like Azure AI Search / Pinecone).
• Data Transformation & CI/CD Tools: Working knowledge of dbt, Infrastructure-as-Code (Terraform), and automated CI/CD pipelines (Azure DevOps / GitHub Actions).
• BI Optimization: Hands-on experience optimizing Direct Lake semantic models and Delta tables for fast Power BI reporting.

Education &amp- Certifications

Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related quantitative field.

Preferred Certifications (Added Advantage)

• o Microsoft Certified: Fabric Data Engineer Associate (DP-700)