Sr AI Data Engineer

Keka Inc.; Hyderabad; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Keka Inc. is seeking a senior Data Engineer to architect and maintain data pipelines, data models and governance across enterprise systems. You will leverage Databricks, PySpark and Python to deliver scalable data platforms, working with business stakeholders to translate needs into robust solutions.

Ideal candidates bring 7+ years in data engineering, strong SQL and experience with ML-enabled data workflows, including agent-based processing and secure access controls.

Qualifications

• Proven 7+ years in data engineering with big data platforms.
• Hands-on with Databricks and PySpark in production.
• Strong SQL development and data modelling capabilities.
• Experience building end-to-end data ingestion and transformation pipelines.
• Knowledge of data governance, quality, lineage and reconciliation.

Responsibilities

• Design and implement data pipelines and warehouse solutions.
• Collaborate with business and tech teams to translate requirements.
• Develop AI-powered data engineering solutions and agent workflows.

Skills

Databricks

PySpark

SQL development

Dimensional modelling

Star schemas

SCD

Data pipelines

AI agent development

Python

Data governance

Tools

Databricks

PySpark

SQL

Microsoft Fabric

LangChain

LangGraph

Python

Job description

• 7+ years of experience in Data Engineering.
• Minimum 3 years of hands-on experience with Databricks.
• Strong hands-on experience with:
• PySpark
• Databricks Workflows
• Unity Catalog
• Strong SQL development and optimization skills.
• Strong experience in dimensional data modelling and star schemas.
• Hands-on experience with Slowly Changing Dimensions (SCD).
• Experience building production-grade data ingestion and transformation pipelines.
• Experience integrating multiple enterprise source systems, preferably ERP, planning and HR systems.
• Hands-on experience integrating Databricks with Microsoft Fabric.
• Good understanding of:
• Microsoft OneLake
• Fabric Lakehouses
• OneLake shortcuts
• Proven hands-on experience building LLM/AI agents using Python.
• Proven hands-on experience building MCP servers using Python in a real-world project.
• Understanding of secure and governed access for AI agents and tools.
• Strong understanding of data quality, reconciliation, lineage and data governance.
• Ability to work with business and technical stakeholders to translate requirements into data solutions.

Strongly Preferred Skills

• Experience in Finance, FP&A or financial data engineering.
• Strong understanding of:
• GL structures
• Budgets
• Forecasts
• Allocations
• Experience with TBM (Technology Business Management) classification or similar financial classification frameworks.
• Experience building AI-powered data engineering or ingestion solutions.
• Experience with agentic workflows for data mapping, classification and anomaly detection.
• Experience with additional LLM frameworks such as LangChain, LangGraph or similar.
• Experience with enterprise AI governance and secure AI/data access.
• Experience with Databricks data governance and lineage.
• Experience implementing automated data reconciliation and financial controls.
• Experience working with Microsoft Fabric-based enterprise analytics environments.