Databricks Architect

Genpact Ltd.; Pune District, Bengaluru, Delhi; Ibrido; INR 1.800.000 - 3.000.000; Tempo pieno

Descrizione del lavoro

Genpact Ltd. is seeking a Databricks Architect to design and implement end-to-end data pipelines, blending batch and streaming processes across Azure/AWS/GCP.

The role emphasizes deep Data Engineering expertise, Unity Catalog governance, and Databricks Lakehouse implementation in enterprise environments. You will collaborate with senior architects and engineers to deliver scalable, cost-efficient data platforms, including Delta Lake, DBT integrations, and robust testing.

Competenze

• Must have experience in Data Engineering domain.
• Must have implemented at least 4 project end-to-end in Databricks.
• Must have skills: Azure data factory, Azure data bricks, Python and Pyspark.
• Expert with database technologies and ETL tools.
• Hands-on experience on designing and developing scripts for custom ETL processes and automation in Azure data factory, Azure databricks, Python, Pyspark etc.
• Good knowledge of AZURE, AWS, GCP Cloud platform services stack.
• Hands-on experience on designing and developing scripts for custom ETL processes and automation in Azure data factory, Azure databricks, Delta lake, Databricks workflows orchestration, Python, Pyspark etc.
• Good Knowledge on Unity Catalog implementation.
• Good Knowledge on integration with other tools like DBT, other transformation tools.
• Good knowledge on Unity Catalog integration with Snowlflake
• Must be well versed with Databricks Lakehouse concept and its implementation in enterprise environments.
• Must have good understanding to create complex data pipeline
• Must have good knowledge of Data structure & algorithms.
• Must be strong in SQL and sprak-sql.
• Must have strong performance optimization skills to improve efficiency and reduce cost.
• Must have worked on both Batch and streaming data pipeline.
• Must have extensive knowledge of Spark and Hive data processing framework.
• Must have worked on any cloud (Azure, AWS, GCP) and most common services like ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, Cloud databases.
• Must be strong in writing unit test case and integration test
• Must have strong communication skills and have worked on the team of size 5 plus
• Must have great attitude towards learning new skills and upskilling the existing skills. Preferred Qualifications
• Good to have Unity catalog and basic governance knowledge.
• Good to have Databricks SQL Endpoint understanding.
• Good To have CI/CD experience to build the pipeline for Databricks jobs.

Mansioni

• Maintains close awareness of new and emerging technologies and their potential application for service offerings and products.
• Work with architect and lead engineers for solutions to meet functional and non functional requirements.
• Demonstrated knowledge of relevant industry trends and standards.
• Demonstrate strong analytical and technical problem-solving skills.
• Must have experience in Data Engineering domain.

Conoscenze

Databricks

Data Engineering

Python

Pyspark

SQL

Spark SQL

Delta Lake

Unity Catalog

Azure

AWS

GCP

DBT

CI/CD

ETL

Streaming

Batch processing

Strumenti

Azure Data Factory

Azure Databricks

Databricks Workflows

Delta Lake

Snowflake

DBT

CosmosDB

DynamoDB

S3

ADLS

Descrizione del lavoro

Role & responsibilities

In this role, the Databricks Architect is responsible for solving the real-world cutting-edge problem to meet both functional and non-functional requirements.

• Maintains close awareness of new and emerging technologies and their potential application for service offerings and products.
• Work with architect and lead engineers for solutions to meet functional and non functional requirements.
• Demonstrated knowledge of relevant industry trends and standards.
• Demonstrate strong analytical and technical problem-solving skills.
• Must have experience in Data Engineering domain.

Preferred candidate profile

• Must have experience in Data Engineering domain .
• Must have implemented at least 4 project end-to-end in Databricks.
• Must have at least experience on databricks which consists of various components as below
• Must have skills: Azure data factory, Azure data bricks, Python and Pyspark
• Expert with database technologies and ETL tools.
• Hands-on experience on designing and developing scripts for custom ETL processes and automation in Azure data factory, Azure databricks, Python, Pyspark etc.
• Good knowledge of AZURE, AWS, GCP Cloud platform services stack
• Hands-on experience on designing and developing scripts for custom ETL processes and automation in Azure data factory, Azure databricks, Delta lake, Databricks workflows orchestration, Python, Pyspark etc.
• Good Knowledge on Unity Catalog implementation.
• Good Knowledge on integration with other tools like DBT, other transformation tools.
• Good knowledge on Unity Catalog integration with Snowlflake
• Must be well versed with Databricks Lakehouse concept and its implementation in enterprise environments.
• Must have good understanding to create complex data pipeline
• Must have good knowledge of Data structure & algorithms.
• Must be strong in SQL and sprak-sql.
• Must have strong performance optimization skills to improve efficiency and reduce cost.
• Must have worked on both Batch and streaming data pipeline.
• Must have extensive knowledge of Spark and Hive data processing framework.
• Must have worked on any cloud (Azure, AWS, GCP) and most common services like ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, Cloud databases.
• Must be strong in writing unit test case and integration test
• Must have strong communication skills and have worked on the team of size 5 plus
• Must have great attitude towards learning new skills and upskilling the existing skills. Preferred Qualifications

Preferred Qualifications

• Good to have Unity catalog and basic governance knowledge.
• Good to have Databricks SQL Endpoint understanding.
• Good To have CI/CD experience to build the pipeline for Databricks jobs.