Azure Data Engineer

GP Strategies; Chennai District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

GP Strategies in Chennai-Hybrid is seeking a Software Integrations Developer to architect connectivity across our digital ecosystem. You will design, develop, and maintain robust integration solutions using the Azure stack, enabling seamless data flow and scalable processing.

Responsibilities include building end-to-end data pipelines, ETL/ELT workflows, and Databricks-based architectures. You will collaborate with stakeholders, implement secure APIs, and optimize performance across the data

Qualifications

• 5+ years of Data Engineering / Data Platform development experience.

Responsibilities

• Design end-to-end data pipelines (batch and streaming) using Azure Data Factory / Synapse Pipelines and Databricks.
• Develop scalable ETL/ELT workflows using PySpark, SQL, and Databricks notebooks.
• Build and optimize data ingestion frameworks for structured and unstructured data from multiple sources.
• Lead the design and implementation of Databricks-based lakehouse architecture with Bronze/Silver/Gold layers.
• Optimize Spark jobs for performance, scalability, and cost.
• Implement data versioning, incremental processing, and governance standards.
• Utilize Azure services (Data Factory, ADLS Gen2, Synapse, Key Vault) for secure scalable solutions.
• Design, build, and maintain integrations using Azure Functions, Logic Apps, Service Bus/Event Grid.
• Develop and expose secure REST APIs and event-driven endpoints via Azure API Management.
• Implement event-driven, near-real-time messaging with Event Hub/Grid/Service Bus.
• Monitor and optimize data workflows for reliability and cost efficiency.
• Manage CI/CD deployments with Azure DevOps and maintain documentation.
• Collaborate with stakeholders to translate requirements into scalable data solutions.

Skills

Data engineering

Strong communication

Agile/Scrum

Problem solving

Tools

Azure Databricks

Azure Data Factory

Azure Data Lake Storage

Azure Synapse Analytics

Python (PySpark)

SQL

REST APIs

Azure Functions

Logic Apps

Service Bus / Event Grid

Azure API Management

Databricks Delta Lake

Job description

Job Title: Software Integrations Developer

Location: Chennai-Hybrid Schedule

Permanent role

Role Purpose

The Software Integrations Developer will be the architect of connectivity within our digital ecosystem. You will design, develop, and maintain robust integration solutions that bridge the gap between disparate systems, ensuring seamless data flow and high-performance processing. By leveraging the Microsoft Azure stack, you will transform complex business requirements into scalable, automated integration workflows that drive organizational efficiency.

Key Accountabilities

• Data Engineering & Pipeline Development: Design and implement end-to-end data pipelines (batch and streaming) using Azure Data Factory / Synapse Pipelines and Databricks.
• ETL/ELT Development: Develop scalable ETL/ELT workflows using PySpark, SQL, and Databricks notebooks to process large-scale datasets efficiently.
• Data Ingestion: Build and optimize data ingestion frameworks for structured and unstructured data from multiple enterprise and external sources.
• Databricks & Lakehouse Architecture: Lead the design and implementation of Databricks-based lakehouse architecture, including Delta Lake and medallion (bronze, silver, gold) data layers.
• Spark Optimization: Optimize Spark jobs for performance, scalability, and cost efficiency, including partitioning, caching, and cluster tuning.
• Data Governance & Processing: Implement data versioning, incremental processing, and ensure high data quality, consistency, and governance standards.
• Azure Data Platform: Utilize Azure services such as Data Factory, Data Lake Storage, Synapse Analytics, and Key Vault to build secure, scalable data solutions.
• Application & Systems Integration: Design, build, and maintain integration solutions using Azure Functions, Logic Apps, and Service Bus / Event Grid to connect line-of-business applications, SaaS platforms, and external services with the data platform.
• API Development & Management: Develop and expose secure, well-documented REST APIs and event-driven endpoints, handling publishing, versioning, throttling, and authentication through Azure API Management.
• Event-Driven & Real-Time Messaging: Implement reliable, decoupled event-driven and near-real-time integration patterns using Event Hub, Event Grid, and Service Bus for resilient message and data exchange across systems.
• Performance & Optimization: Monitor and continuously optimize data workflows for reliability, performance, and cost efficiency.
• DevOps & Governance: Manage code repositories, CI/CD pipelines, and deployments using Azure DevOps, ensuring high-quality documentation and best practices.
• Collaboration & Stakeholder Engagement: Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions and provide technical leadership.

What Success Looks Like

Reliable and scalable Azure integration architectures with 99.9% uptime, optimized data throughput, and minimal technical debt; clear technical and API documentation; strict adherence to security governance; and seamless data flow that earns positive feedback from business stakeholders.

Candidate Profile

Experience & Capability

• 5+ years of experience in Data Engineering / Data Platform development.
• 3+ years of hands-on experience with Azure Databricks.
• Strong experience with Azure Data Factory, ADLS Gen2, and Synapse Analytics.
• Proven experience in building enterprise-scale data pipelines.
• Hands-on experience building application and API integrations using Azure Functions, Logic Apps, and Azure API Management.
• Experience working in Agile/Scrum environments and full SDLC lifecycle.

Technical Strength

• Development: Strong proficiency in Python (PySpark), SQL, and experience working with Databricks notebooks for scalable data processing and transformation logic.
• Data Design: Deep knowledge of ETL/ELT design patterns, including Slowly Changing Dimensions (SCD), medallion architecture (bronze, silver, gold), and data modelling for analytics.
• Platforms & Tools: Expert-level experience with Azure Data Factory / Synapse Pipelines, Azure Data Lake Storage, and Azure Databricks including Delta Lake.
• Integration & APIs: Hands-on experience designing integration and API solutions with Azure Functions, Logic Apps, Service Bus / Event Grid, and Azure API Management, including REST API design and secure authentication (OAuth 2.0 / managed identities).
• Databricks Expertise: Advanced hands-on experience with Spark optimization, cluster management, job orchestration, and performance tuning in Databricks.
• Bonus Strengths: Experience with streaming technologies (Event Hub/Kafka), Microsoft Fabric, CI/CD for data platforms, or Power BI will be considered a significant advantage.

Mindset & Behaviours

• Design-first and quality-focused mindset.
• Comfortable working in ambiguity and evolving requirements.
• Strong communicator able to explain technical concepts to non-technical audiences.
• Proactive, collaborative, and continuous improvement oriented.
• Demonstrates strong sense of urgency and ownership.