Azure Data Engineer Lead/Architect

Creative Synergies Group; Bengaluru; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Creative Synergies Group is looking for an Azure Data Engineer Lead/Architect in Bengaluru, India. The role involves designing, implementing, and optimizing scalable data solutions on the Microsoft Azure platform and requires 8+ years of data engineering experience with strong expertise in Azure technologies, particularly Azure Databricks and Azure Data Factory. Responsibilities include mentoring a team, building data pipelines, and ensuring compliance and data governance standards are met.

Qualifications

• 8+ years of experience in data engineering with 5+ years in Microsoft Azure cloud technologies.
• Strong experience in building and optimising large-scale data pipelines in a cloud environment.
• Hands-on experience with data security and compliance standards.

Responsibilities

• Lead the design and implementation of scalable data solutions on Azure.
• Build and optimize end-to-end data pipelines using Azure services.
• Implement best practices for data governance, security, and compliance.

Skills

Azure Databricks

Data Engineering

Cloud Technologies

Python (PySpark)

SQL

Data Governance

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field

Tools

Azure Data Factory

Azure Synapse Analytics

Azure SQL Database

Azure Blob Storage

Azure Data Lake

Spark

Hadoop

Kafka

Job description

Job Summary

We are seeking a skilled and motivated Azure Data Engineer Lead/Architect with strong relevant experience. Candidate will play a pivotal role in designing, implementing, and optimizing large‑scale data solutions on the Microsoft Azure platform. Candidate will work closely with other cross‑functional teams to ensure data solutions are scalable, secure, and high‑performance. The ideal candidate will have a strong background in data engineering, deep expertise in Azure Databricks and proven leadership skills to guide and mentor the team.

Key Responsibilities

• Architect and Design Data Solutions: Lead the design and implementation of scalable, secure and high-performing data solutions on Azure, ensuring alignment with business goals and technical requirements.
• Data Pipeline Development: Build and optimize end-to-end data pipelines to collect, process and analyze data from various sources using Azure Databricks, Azure Data Factory and other Azure data services.
• Cloud Infrastructure Management: Architect and maintain cloud data infrastructure using Azure Data Lake, Azure Synapse Analytics, Azure SQL Data Warehouse and Azure Blob Storage.
• Collaborate with Cross-functional Teams: Work closely with data architects, business analysts and developers to gather requirements, define data models and ensure smooth integration of data across systems.
• Design and build scalable data pipelines to support AI/ML use cases, including feature engineering, model training datasets, and real‑time inference workflows using Azure-native services.
• Collaborate with Data Scientists and ML Engineers to productionise machine learning models by implementing robust data workflows, feature stores, model monitoring pipelines, and MLOps best practices on Azure (e.g., integration with Azure ML, CI/CD for ML pipelines).
• Develop and optimise data architectures to support advanced analytics, AI, and Generative AI workloads, ensuring data quality, governance, lineage, and compliance across structured and unstructured data sources.
• Performance Tuning and Optimization: Continuously monitor and optimise the performance of data workflows and pipelines to ensure efficient data processing.
• Data Governance & Security: Implement best practices for data governance, security and compliance, ensuring that data solutions adhere to company and regulatory standards.
• Mentor and Lead Teams: Provide technical leadership and mentorship to junior data engineers, guiding them in the best practices of data engineering and cloud technologies.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in data engineering, with at least 5+ years of hands‑on experience with Microsoft Azure cloud technologies.
• Strong experience with Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, Azure Data Lake and other related Azure services.
• Hands‑on experience using these services: PySpark, Notebooks, Structured Streaming, Unity Catalog, Delta Live Tables, Workflows and SQL Warehouse.
• In‑depth knowledge of data modelling, ETL/ELT processes, data warehousing and data pipeline architecture.
• Proven experience in building and optimising large‑scale data pipelines in a cloud environment.
• Strong experience with SQL, NoSQL databases and big‑data processing technologies (e.g., Spark, Hadoop, Kafka).
• Hands‑on experience with data transformation, data integration and data quality frameworks.
• Expertise in data security and compliance standards in a cloud‑based environment.
• Ability to design and implement data governance strategies for large datasets.
• Strong leadership, communication and collaboration skills, with a proven ability to guide teams and manage complex projects.