Azure Databricks Engineer (PySpark)

PSRTEK; Hyderabad; On-site; INR 900,000 - 1,300,000; Full time

Job summary

PSRTEK seeks an Azure Databricks Engineer (PySpark) to design, build, and optimize scalable data pipelines on Azure. You will work with large distributed data systems and support analytics initiatives, collaborating with data scientists and stakeholders.

The role focuses on developing ETL/ELT pipelines, ensuring data quality and security, and optimizing Spark workloads for performance and cost efficiency within a cloud data platform.

Qualifications

• Proficient in PySpark and Azure Databricks
• Strong SQL skills and data warehousing understanding
• Experience with distributed computing concepts and Git version control

Responsibilities

• Develop and maintain ETL/ELT pipelines using PySpark on Azure Databricks
• Design scalable data solutions using Apache Spark
• Integrate data from sources including Azure Data Lake Storage and SQL databases
• Optimize Spark jobs for performance and cost efficiency
• Collaborate with data scientists and analysts to ensure data quality and governance
• Monitor and troubleshoot production data pipelines

Skills

PySpark

Azure Databricks

SQL

Distributed computing

Azure Data Lake Storage

Data warehousing

Git

Job description

Job Title: Azure Databricks Engineer (PySpark)

Job Summary

We are seeking a skilled Azure Databricks Engineer with strong expertise in PySpark to design, build, and optimize scalable data pipelines on the Azure cloud platform. The ideal candidate will work with large-scale distributed data systems and support advanced analytics and data engineering initiatives.

Key Responsibilities

• Develop and maintain ETL/ELT pipelines using PySpark on Azure Databricks
• Design scalable data solutions using Apache Spark
• Integrate data from multiple sources (Azure Data Lake, SQL DBs, APIs, streaming sources)
• Optimize Spark jobs for performance and cost efficiency
• Work with Azure Data Factory for orchestration
• Implement data transformation, cleansing, and validation logic
• Collaborate with data scientists, analysts, and stakeholders
• Ensure data quality, governance, and security standards
• Monitor and troubleshoot production data pipelines

Required Skills

• Strong experience with PySpark
• Hands-on experience with Azure Databricks
• Good understanding of SQL
• Experience with distributed computing concepts
• Familiarity with Azure Data Lake Storage
• Knowledge of data warehousing concepts
• Experience with version control (Git)

Preferred Skills

• Experience with Delta Lake
• Knowledge of streaming frameworks (Spark Streaming / Kafka)
• Experience with Azure Synapse Analytics
• Python programming beyond PySpark
• CI/CD pipelines (Azure DevOps)