Data Engineer

InfoVision Inc.; Pune District; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

InfoVision Inc. invites an experienced Senior Data Engineer to design and manage enterprise data pipelines on Microsoft Azure and Databricks. You will own schema design, API development, and data infrastructure for analytics and intelligence products.

Responsibilities include building scalable data workflows, relational/non-relational DB schema design, and RESTful API development with FastAPI. Collaboration across engineering teams and clear stakeholder communication are essential.

Qualifications

• 4+ years data engineering or data platform experience.
• Relational and non-relational database expertise.
• MongoDB: advanced schema design, aggregation, indexing, sharding, performance optimization.
• Azure Cosmos DB: multi-model APIs, partitioning, consistency, throughput management.
• Graph databases: Cosmos DB Gremlin API, graph modeling, traversals, relationship analytics.

Responsibilities

• Design, build, manage enterprise data pipelines on Azure and Databricks.
• Own schema design, API development, and data infrastructure for analytics and intelligence products.
• Design relational and non-relational database schemas and optimize performance.
• Build RESTful APIs using FastAPI with async endpoints and OpenAPI documentation.
• Collaborate with multiple engineering teams and communicate data insights to stakeholders.

Tools

MongoDB

Azure Cosmos DB

Cosmos Gremlin API

FastAPI

Azure API Management (APIM)

Databricks

Azure Data Factory

Azure Databricks

Azure Cosmos DB multi-region

Synapse/Azure ecosystem

Job description

Senior Data Engineer – Data Pipelines & Cloud Databases

Location: Pune | Experience: 4+ years | Type: Full-Time

Role Overview

Design, build, manage enterprise data pipelines on Azure and Databricks. Own schema design, API development, and data infrastructure for analytics and intelligence products.

Key Responsibilities

Data Pipeline Development

• Implement robust, scalable data pipelines using Microsoft Azure and Databricks stack
• Build reusable data pipeline components and frameworks
• Design and optimize data workflows for performance and reliability
• Monitor pipeline health and implement automated alerting

Database Architecture

• Design relational and non-relational database schemas
• MongoDB: schema design, aggregation pipelines, indexing, sharding, replica sets, performance tuning
• Azure Cosmos DB: multi-model access, partitioning, consistency levels, throughput management (RU/s), global distribution
• Azure Cosmos DB Gremlin API: graph data modeling, traversals, vertex/edge design, relationship analytics

API Development

• Build RESTful APIs using FastAPI with async endpoint design
• Implement Pydantic models, middleware, dependency injection, background tasks
• Generate and maintain auto-generated OpenAPI/Swagger documentation
• Configure Azure API Management (APIM) for API gateway and security

Project & Stakeholder Management

• Collect progress updates from squads regularly
• Consolidate updates into weekly project status reports
• Develop L3-level project plans with task details, milestones, dependencies
• Participate in early-stage design and feature definition
• Communicate complex data insights to non-technical stakeholders

Collaboration & Integration

• Work across multiple engineering teams on prototype integration
• Support integration of proven prototypes into core intelligence products
• Strong team collaboration and cross-functional communication
• Knowledge sharing and documentation

Required Experience

Data Engineering & Databases

• 4+ years data engineering or data platform experience
• Relational and non-relational database expertise
• MongoDB: advanced schema design, aggregation, indexing, sharding, performance optimization
• Azure Cosmos DB: multi-model APIs, partitioning, consistency, throughput management
• Graph databases: Cosmos DB Gremlin API, graph modeling, traversals, relationship analytics

API & Backend Development

• FastAPI proficiency: async endpoints, Pydantic models, middleware, dependency injection
• Background tasks and job scheduling
• RESTful API design and best practices
• OpenAPI/Swagger documentation

Cloud Platform

• Azure fundamentals and hands-on experience
• Azure Data Factory or Databricks for ETL/ELT
• Azure Cosmos DB multi-region setup
• Azure API Management (APIM) configuration

Data Pipeline Skills

• ETL/ELT pipeline design and development
• Data quality validation and monitoring
• Schema design for analytics and reporting
• Performance optimization and scalability

Preferred Experience

• Databricks Delta Lake experience
• Azure Synapse Analytics
• Python for data engineering
• Spark SQL optimization
• Real-time data streaming
• Data governance and metadata management
• Agile/Scrum development model
• CI/CD pipeline experience