Azure Databricks Engineer

Maddisoft Solutions; Hyderabad; Remote; INR 2,500,000 - 4,500,000; Full time

Job summary

Maddisoft Solutions in Hyderabad (Remote) is seeking an Azure Databricks Engineer with 8-12 years of data engineering experience to design and build scalable cloud data pipelines.

You will work with Azure Databricks, PySpark, Python, SQL, Azure Data Factory, and ADLS Gen2, developing Delta Lake tables and performing transformations. The role emphasizes collaboration with cross-functional teams, adherence to data governance, and optimizing Spark workloads for cost and performance.

Qualifications

• 8-12 years of experience in Data Engineering
• Hands-on experience with Azure Databricks
• Strong programming experience in Python/PySpark
• Advanced SQL skills with performance optimization
• Experience with Azure Data Factory (ADF)
• Experience with Azure Data Lake Storage Gen2 (ADLS)
• Strong understanding of Delta Lake and lakehouse architecture
• Experience developing and managing ETL/ELT data pipelines
• Knowledge of data modeling, data integration, and data quality concepts
• Experience with Git and CI/CD practices
• Strong troubleshooting and problem-solving skills

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark
• Develop complex data transformations using Python, PySpark, and SQL
• Build and optimize ETL/ELT pipelines using Azure Data Factory (ADF)
• Work with ADLS Gen2 for data ingestion, storage, and processing
• Develop and maintain Databricks notebooks, jobs, workflows, and Delta Lake tables
• Implement data cleansing, transformation, validation, and aggregation processes
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency
• Work with structured and semi-structured data from various sources
• Implement incremental data loading and data processing strategies
• Troubleshoot data pipeline failures and resolve data-quality issues
• Collaborate with data architects, analysts, application teams, and business stakeholders
• Follow standards for data security, governance, monitoring, and documentation
• Participate in code reviews, testing, deployment, and production support activities

Skills

Azure Databricks

Python/PySpark

SQL

Azure Data Factory

ADLS Gen2

Delta Lake

ETL/ELT pipelines

Git/CI-CD

Data modeling

Data quality

Security & governance

Education

Bachelors degree in CS/IT/Engineering

Tools

Azure Synapse Analytics

Power BI

Job description

Job Title: Azure Databricks Engineer
Location: Hyderabad (Remote)

Job Summary

We are seeking a skilled Azure Databricks Engineer with 812 years of experience in data engineering and cloud-based data platforms. The ideal candidate will have hands-on experience with Azure Databricks, PySpark, SQL, Azure Data Factory, and Azure Data Lake Storage and will be responsible for developing scalable data pipelines and supporting data integration and transformation initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Develop complex data transformations using Python, PySpark, and SQL.
• Build and optimize ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Azure Data Lake Storage Gen2 (ADLS) for data ingestion, storage, and processing.
• Develop and maintain Databricks notebooks, jobs, workflows, and Delta Lake tables.
• Implement data cleansing, transformation, validation, and aggregation processes.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Work with structured and semi-structured data from various sources.
• Implement incremental data loading and data processing strategies.
• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Collaborate with data architects, analysts, application teams, and business stakeholders.
• Follow established standards for data security, governance, monitoring, and documentation.
• Participate in code reviews, testing, deployment, and production support activities.

Required Skills

• 8 - 12 years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks.
• Strong programming experience in Python/PySpark.
• Advanced SQL skills, including complex queries and performance optimization.
• Experience with Azure Data Factory (ADF).
• Experience working with Azure Data Lake Storage Gen2 (ADLS).
• Strong understanding of Delta Lake and lakehouse architecture.
• Experience developing and managing ETL/ELT data pipelines.
• Knowledge of data modeling, data integration, and data quality concepts.
• Experience with Git and standard CI/CD practices.
• Strong troubleshooting and problem-solving skills.

Preferred Skills

• Experience with Azure Synapse Analytics.
• Knowledge of Microsoft Fabric is a plus.
• Experience with Azure DevOps and CI/CD pipelines.
• Knowledge of Databricks Unity Catalog and data governance.
• Experience working with streaming technologies such as Kafka or Azure Event Hubs.
• Familiarity with Power BI and analytical data platforms.
• Experience with automated testing and data validation frameworks.
• Databricks or Azure certifications are a plus.

Education

Bachelors degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent professional experience.

Ideal Candidate

The ideal candidate is a hands-on data engineer who can independently develop and troubleshoot Azure Databricks data pipelines, work effectively with cross-functional teams, and contribute to building reliable and scalable cloud data solutions.