DataBricks - Lead Programmer Analyst

Bilvantis Technologies; Hyderabad; On-site; INR 3,000,000 - 5,400,000; Full time

Job summary

Bilvantis Technologies in Hyderabad seeks a DataBricks Lead Programmer Analyst to design and maintain scalable ETL/ELT data pipelines on the Lakehouse platform. You will build Spark jobs (PySpark/Scala), manage Delta Lake tables, and optimize performance across batch and streaming workloads.

You will collaborate with data scientists and analysts, implement governance with Unity Catalog, and orchestrate workflows using Databricks Workflows or Airflow.

Qualifications

• 5–7 years of Data Engineering experience.
• Expert-level PySpark (Scala a plus) and SQL.
• Delta Lake and Lakehouse/medallion architecture knowledge.
• Cloud data services (AWS/Azure/GCP) and data workflows.
• CI/CD, Git, and DevOps for data engineering.
• Strong data governance, Unity Catalog, and security understanding.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines on Databricks Lakehouse.
• Build and optimize Spark jobs for batch and streaming data.
• Manage Delta Lake tables with partitioning, schema evolution, and performance tuning.
• Orchestrate workflows with Databricks Workflows, DLT, and Airflow.
• Implement medallion architecture and integrate cloud data services.
• Collaborate with analysts/scientists and ensure data quality and observability.

Skills

Data engineering exp

Spark PySpark

SQL proficiency

Delta Lake

Cloud platforms

Workflow orchestration

Data modeling & ETL/ELT

Python for data eng

Git/DevOps for data

Unity Catalog & security

Communication skills

Education

Bachelor's or Master's in Computer Science/Engineering

Tools

Databricks

Airflow

Terraform

Jira

Confluence

Job description

DataBricks - Lead Programmer Analyst

We are looking for a highly self-motivated individual with DataBricks development as a Lead Programmer Analyst:

Required Skills & Qualifications

• Experience should have 5 to 7 Years of Data Engineering.
• Expert-level Apache Spark skills using PySpark (Scala a plus).
• Strong proficiency in SQL for data transformation and performance tuning.
• Solid experience with Delta Lake and the Lakehouse/medallion architecture.
• Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its data services.
• Experience with workflow orchestration (Databricks Workflows, Delta Live Tables, or Airflow).
• Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
• Experience with Python for data engineering and automation.
• Familiarity with CI/CD, Git, and DevOps practices for data.
• Good understanding of data governance, security, and Unity Catalog.
• Strong problem-solving and communication skills.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Good To Have

• Databricks Certified Data Engineer Associate or Professional certification.
• Experience with streaming technologies (Structured Streaming, Kafka, Event Hubs, Pub/Sub).
• Exposure to machine learning workflows and MLflow.
• Experience with Infrastructure-as-Code (Terraform) and cloud cost optimization.
• Experience working in Agile delivery environments.
• Experience in data warehouse design and maintenance
• Experience in agile development processes using Jira and Confluence.
• Experience in cross-functional teams.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines on the Databricks Lakehouse Platform.
• Build and optimize Apache Spark (PySpark/Scala) jobs for batch and streaming data processing.
• Implement and manage Delta Lake tables, including partitioning, schema evolution, and performance tuning (Z-ordering, caching, file compaction).
• Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and job scheduling tools such as Airflow.
• Implement the medallion architecture (bronze, silver, gold layers) for reliable, governed data.
• Integrate Databricks with cloud data services (AWS, Azure, or GCP) and source systems.
• Apply data governance and security using Unity Catalog, access controls, and lineage.
• Optimize cluster configuration, cost, and performance across workloads.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver trusted datasets.
• Ensure data quality, testing, monitoring, and observability across pipelines.
• Contribute to CI/CD practices for data engineering (version control, automated deployment).