Lead/ Senior Azure Data Engineer_ Exp: 8+ Years

Atyeti; Mumbai; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Atyeti is seeking an experienced Data Engineering Lead to drive Azure-based data platforms, pipelines, and governance. You will mentor a team, migrate on-prem workloads to the cloud, and collaborate with stakeholders to deliver scalable data solutions.

The role emphasizes hands-on PySpark development, API creation on Azure Function Apps, and optimization of data flows for business insights via Power BI.

Qualifications

• Leadership experience of data engineering teams.

Responsibilities

• Lead data engineering projects end-to-end.
• Guide team aligning with architectural principles.
• Migrate on-prem data warehouses to Azure.
• Design data pipelines with ADF/Databricks.
• Build PySpark-based data processing.
• Develop APIs on Azure Function Apps.
• Leverage Azure Logic Apps for workflows.
• Implement lakehouse/data warehouse architectures.
• Plan capacity and tune ADF/Databricks pipelines.
• Support Power BI visualizations.
• Participate in full SDLC including testing and deployment.
• Experience in relational and dimensional modelling across big data.
• Setup CI/CD for Azure services via Azure DevOps.
• Work in secured Azure environments using Key Vault and Managed Identities.
• Engage with business for requirements and resolution of queries.
• Coordinate offshore teams and contribute to project lifecycle.
• Promote Azure Data skills through knowledge sharing.

Skills

Azure Data Factory

Azure Databricks

PySpark

Azure SQL

Azure Function Apps

Azure Logic Apps

CI/CD

Stakeholder mgmt

Tools

Power BI

Azure DevOps

Azure Key Vault

Managed Identities

Job description

Role & responsibilities

• Lead data engineering projects to build and operationalize data solutions for business using Azure services in combination with custom solutions Azure Data Factory, Azure Data Flows, Azure Databricks, Azure Data Lake Gen 2, Azure SQL etc
• Proven experience on leading a team of data engineers providing technical guidance and ensuring alignment with agreed architectural principles
• Experience in migrating on-premise data warehouses to data platforms on AZURE cloud
• Designing and implementing data engineering, ingestion and transformation functions using ADF, Databricks
• Proficient in Py-Spark
• Experience in building Python based APIs on Azure Function Apps
• Experience on Azure Logic apps
• Experience in Lakehouse/Datawarehouse implementation using modern data platform architecture
• Capacity Planning and Performance Tuning on ADF & Databricks pipelines
• Support data visualization development using Power BI
• Exposure across all the SDLC process, including testing and deployment
• Experience in relational and dimensional modelling, including big data technologies
• Experience in Azure DevOps – Build CI/CD pipelines for ADF, ADLS, Databricks, Azure SQL DB etc
• Experience of working in secured Azure environments using Azure KeyVaults, Service Principals, and Managed Identities
• Good to have knowledge on Apigee (Googles API Management)
• Understanding of data masking, encryption and other practices used in handling sensitive data
• Ability to interact with Business for requirement gathering and query resolutions
• Working on off shore office based development teams, collaborating within a team environment and participating in typical project lifecycle activities such as requirement analysis, testing and release
• Develop Azure Data skills within the team through knowledge sharing sessions, articles, etc.
• Adherence to organisations Risk & Controls requirements
• Should have skills for Stakeholder management, process adherence, planning & documentationss