Azure DataBricks,PySpark

Kaizen Sra Technologies; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Kaizen Sra Technologies seeks a data engineer with 6+ years of experience to design and optimize ETL/ELT pipelines using Azure Databricks, PySpark, and SQL. You will build medallion data layers with Delta Lake, implement batch and streaming pipelines, and ensure data quality and governance across ADLS Gen2 and Synapse.

Candidate should have hands-on Azure Databricks, Spark, Python, and SQL, plus experience with Git, CI/CD, and Azure data services.

Qualifications

• 6–8 years of experience in data engineering, ETL development, or big data processing.
• Strong hands-on experience with Azure Databricks, Spark, and PySpark.
• Strong programming skills in Python and SQL; experience with Azure Data Lake Gen2.
• Experience developing pipelines using Azure Data Factory.
• Knowledge of Delta Lake and lakehouse architecture.
• Experience with relational and NoSQL databases.
• Understanding of data modeling, ETL/ELT concepts, and data warehousing.
• Experience with Git-based source control and CI/CD practices.
• Familiarity with Azure services such as Azure Key Vault, Synapse Analytics, SQL Database, Event Hubs, and monitoring.

Responsibilities

• Design, develop and optimise scalable ETL/ELT pipelines using Databricks.
• Develop production-grade data transformations using Python, PySpark and SQL.
• Build Bronze/Silver/Gold data layers using Delta Lake and medallion architecture.
• Implement batch and streaming ingestion patterns where required.
• Work with Databricks Workflows and Unity Catalog for orchestration and governance.
• Integrate Databricks with Azure data services including ADLS Gen2 and Synapse.
• Implement data-quality checks, error handling, monitoring and performance optimisation.
• Develop automated deployments using CI/CD and follow software engineering best practices.
• Troubleshoot production data pipelines and ensure reliability and scalability.

Skills

Databricks

Python/PySpark

SQL

Delta Lake

Databricks Workflows

CI/CD

Azure Data Lake Gen2

Unity Catalog

Git

Tools

Azure Databricks

Azure Synapse

Azure Data Factory

ADLS Gen2

Job description

ADB + Pyspark + Kafka

• Design, develop and optimise scalable ETL/ELT pipelines using Databricks.
• Develop production-grade data transformations using Python, PySpark and SQL.
• Build Bronze/Silver/Gold data layers using Delta Lake and medallion architecture.
• Implement batch and, where required, streaming ingestion patterns.
• Work with Databricks Workflows and Unity Catalog for orchestration and governance.
• Integrate Databricks with Azure data services including ADLS Gen2 and Synapse.
• Implement data-quality checks, error handling, monitoring and performance optimisation.
• Develop automated deployments using CI/CD and follow software engineering best practices.
• Troubleshoot production data pipelines and ensure reliability and scalability.

Mandatory skills:

• Strong hands-on Databricks, Python/PySpark and SQL.
• Delta Lake and Databricks Workflows.
• ETL/ELT development and performance tuning.
• ADLS Gen2 and Azure data ecosystem.
• Unity Catalog exposure.
• Git and CI/CD experience.

2) JD :

Skills Azure Databricks+ Pyspark

Exp 6+ years

• Experience and Qualification:6–8 years of experience in data engineering, ETL development, or big data processing.
• Strong hands-on experience with Azure Databricks. Proficiency in Apache Spark and PySpark.
• Strong programming skills in Python and SQL. Experience with Azure Data Lake Storage Gen2.
• Experience developing pipelines using Azure Data Factory.
• Practical knowledge of Delta Lake and lakehouse architecture.
• Experience working with relational and NoSQL databases.
• Strong understanding of data modeling, ETL/ELT concepts, and data warehousing.
• Experience with Git-based source control and CI/CD practices.
• Knowledge of Azure services such as: Azure Key Vault Azure Synapse Analytics Azure SQL Database Azure Event Hubs Azure Monitor Experience with data security, access control, and governance in Azure environments.