Azure Data Engineer/ Architect

Capgemini; Gurugram District, Chennai District, Bengaluru; Hybrid; INR 1,500,000 - 2,700,000; Full time

Job summary

Capgemini India is seeking an experienced Azure cloud data engineer with 7 to 10 years of experience to build and optimize data pipelines on the Azure Cloud Platform. You will design scalable data architectures and enable advanced analytics and machine learning workflows, collaborating with data scientists, analysts, and developers.

The role requires strong Python and PySpark coding, SQL optimization, and hands-on experience with big data tools such as Hadoop, Spark, and Kafka.

Qualifications

• Must have knowledge in Azure Datalake, Azure function, Azure Databricks, Azure data factory, ADLS
• Working knowledge in Azure Devops, Git flow would be an added advantage.
• Collaborate with data scientists, analysts, and application developers to deliver high-quality data solutions.
• Ensure data quality, governance, and security across all pipelines and storage layers.
• Write efficient and maintainable code in Python and PySpark.
• Develop and optimize SQL queries for data extraction and transformation.
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational databases such as Microsoft SQL Server, MySQL, PostgreSQL, Oracle and NoSQL databases such as Hadoop, Cassandra, MongoDB
• Strong analytic skills related to working with structured, semi structured, unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Implement data quality checks and monitoring, ensuring data accuracy and reliability.
• Working knowledge of message queuing, stream processing, and highly scalable big data data stores.
• Strong problem-solving skills with an emphasis on sustainable and reusable development.
• Experience building and optimizing big data data pipelines, architectures and data sets.
• Hands-on experience with CI/CD tools and DevOps practices.

Responsibilities

• Build and optimize data pipelines on Azure to enable analytics and ML workflows.
• Design scalable data architectures for batch and streaming data.
• Ensure data quality, governance, and security across pipelines and storage.
• Collaborate with data scientists, analysts, and application developers.
• Write clean, efficient Python and PySpark code.
• Optimize SQL queries for data extraction and transformation.
• Work with Hadoop, Spark, Kafka and other big data tools.

Skills

Azure Datalake

Azure Databricks

Azure Data Factory

Python

PySpark

SQL

CI/CD

DevOps

Big data

Kafka

Education

Masters or Bachelor's in Computer Science/Engineering

Tools

Azure DevOps

Git

Job description

Experience

Optimally between 7 to 10 years of experience as Azure cloud data engineer responsible for building and optimizing data pipelines, designing scalable data architectures, and enabling advanced analytics and machine learning workflows on Azure Cloud Platform.

Educational qualifications expected

Masters or Bachelor's degree in Computer Science, Engineering, or a related field.

Requirements

• Must have knowledge in Azure Datalake, Azure function, Azure Databricks, Azure data factory, ADLS
• Working knowledge in Azure Devops, Git flow would be an added advantage.
• Collaborate with data scientists, analysts, and application developers to deliver high-quality data solutions.
• Ensure data quality, governance, and security across all pipelines and storage layers.
• Write efficient and maintainable code in Python and PySpark.
• Develop and optimize SQL queries for data extraction and transformation.
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational databases such as Microsoft SQL Server, MySQL, PostGreSQL, Oracle and NoSQL databases such as Hadoop, Cassandra, Mongo dB
• Strong analytic skills related to working with structured, semi structured, unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Implement data quality checks and monitoring, ensuring data accuracy and reliability.
• Working knowledge of message queuing, stream processing, and highly scalable big data data stores.
• Strong problem-solving skills with an emphasis on sustainable and reusable development.
• Experience building and optimizing big data data pipelines, architectures and data sets.
• Hands-on experience with CI/CD tools and DevOps practices.

Experience/Skills

• Azure Datalake, Azure function, Azure Databricks, Azure data factory, ADLS, Azure Devops
• Proficiency in Python and PySpark.
• Strong SQL skills, including query optimization and performance tuning.
• Familiarity with data warehousing, data lake concepts and ETL processes.
• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational databases such as Microsoft SQL Server, MySQL, PostgreSQL, Oracle, and NoSQL databases such as Hadoop, Cassandra, MongoDB.
• Excellent problem-solving skills with an emphasis on sustainable and reusable development.
• Strong communication and collaboration skills.

Additional Information

• Reporting to : Director- Intelligent Insights and Data Strategy
• Travel : Must be willing to be deployed at client locations anywhere in the world for long and short term as well as should be flexible to travel on shorter duration within India and abroad