Azure Subcontractor

Birlasoft; Hyderabad; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Birlasoft in Hyderabad, India seeks an experienced Databricks Developer to design, develop and maintain scalable data pipelines and analytics solutions using Databricks. The ideal candidate will leverage Spark, Python and SQL to support enterprise data integration and analytics initiatives.

You will build ETL/ELT pipelines, Delta Lake tables, and orchestrate workflows while collaborating with analysts and stakeholders to meet data requirements and ensure quality.

Qualifications

• Databricks experience is essential.
• Strong expertise in Spark/PySpark.
• Proficiency in Python.
• Strong SQL skills.
• Experience with ETL/ELT development.
• Experience with Delta Lake.
• Knowledge of data warehousing concepts.
• Experience with Git and version control.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks.
• Develop scalable data processing frameworks using Apache Spark (PySpark).
• Build and optimize batch and streaming data pipelines.
• Integrate data from multiple sources such as databases, APIs, cloud storage, and enterprise applications.
• Implement data transformation, cleansing, and validation processes.
• Develop Delta Lake tables and optimize data storage using partitioning and Z-Ordering.
• Create reusable notebooks, workflows, and jobs in Databricks.
• Monitor and troubleshoot pipeline failures and performance issues.
• Collaborate with business analysts, architects, and stakeholders to understand data requirements.
• Follow coding standards, version control, and CI/CD best practices.
• Participate in code reviews and technical design discussions.

Skills

Databricks

Spark/PySpark

Python

SQL

ETL/ELT development

Delta Lake

Data warehousing concepts

Git/version control

Tools

Airflow

Job description

Area(s) of responsibility

JD, for a Databricks developer`

We are looking for an experienced Databricks Developer to design, develop, and maintain scalable data pipelines and analytics solutions using Databricks. The ideal candidate should have strong expertise in Apache Spark, Python, SQL, and cloud-based data engineering technologies to support enterprise data integration and analytics initiatives.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks.
• Develop scalable data processing frameworks using Apache Spark (PySpark).
• Build and optimize batch and streaming data pipelines.
• Integrate data from multiple sources such as databases, APIs, cloud storage, and enterprise applications.
• Implement data transformation, cleansing, and validation processes.
• Develop Delta Lake tables and optimize data storage using partitioning and Z-Ordering.
• Create reusable notebooks, workflows, and jobs in Databricks.
• Monitor and troubleshoot pipeline failures and performance issues.
• Collaborate with business analysts, architects, and stakeholders to understand data requirements.
• Follow coding standards, version control, and CI/CD best practices.
• Participate in code reviews and technical design discussions.

Technical Skills

Mandatory Skills

• Strong experience with Databricks
• Expertise in Apache Spark / PySpark
• Proficiency in Python
• Strong SQL skills
• Experience with ETL/ELT development
• Experience with Delta Lake
• Knowledge of data warehousing concepts
• Experience with Git and version control

Preferred Skills

• Experience with cloud platforms:
• Microsoft Azure – Azure Databricks, ADLS Gen2, Azure Data Factory
• Amazon Web Services – S3, Glue, EMR
• Google Cloud – BigQuery, Cloud Storage
• Experience with streaming technologies such as Spark Structured Streaming.
• Knowledge of orchestration tools such as Airflow.
• Understanding of data governance and security concepts.

Nice to Have

• Experience with Medallion Architecture (Bronze, Silver, Gold layers).
• Knowledge of Unity Catalog.
• Experience in performance tuning and cost optimization in Databricks.
• Exposure to DevOps and CI/CD pipelines.
• Certification in Databricks or cloud platforms.

Key Competencies

• Data Modeling
• ETL/ELT Development
• Performance Optimization
• Problem Solving
• Team Collaboration
• Agile/Scrum Methodology
• Documentation and Technical Design