Senior Data Engineer (Databricks)

Kani Solutions Inc; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Kani Solutions Inc in Bengaluru is seeking a seasoned Senior Data Engineer with deep Databricks expertise to extend our enterprise data platform. The role focuses on integrating datasets, building scalable pipelines, and developing business dimensions and measures within the Databricks Lakehouse.

You will design and optimize ETL/ELT processes, implement data governance through Unity Catalog, and collaborate across teams to deliver reliable analytics ready for reporting and downstream

Qualifications

• 8+ years of professional experience in Data Engineering.
• 4+ years of practical, hands-on experience with Databricks (required).

Responsibilities

• Architect and build scalable data pipelines leveraging Databricks and PySpark.
• Connect and incorporate additional data sources into the existing Databricks Lakehouse environment.
• Create and refine dimensional models, business dimensions, measures, and curated datasets for analytical use.
• Prepare, transform, and optimize data to support reporting, analytics, and downstream applications.
• Contribute to data quality initiatives, validation activities, monitoring, and performance tuning.
• Investigate and resolve platform-related issues throughout the project.
• Apply Databricks development, deployment, security, and governance best practices across solutions.
• Use Unity Catalog to address data governance and access control requirements.
• Build, package, and deploy solutions through Databricks Asset Bundles.
• Work closely with the project team to convert business and technical requirements into scalable data solutions.
• Establish and manage CI/CD workflows for data engineering components.
• Use GitHub and/or GitLab to manage source code, branching approaches, and peer code reviews.
• Contribute to automated deployment workflows and release management activities.

Skills

Databricks

PySpark

Python

SQL

Unity Catalog

Databricks Asset Bundles

Security & Access Management

ETL/ELT

Dimensional modelling

CI/CD pipelines

GitHub

GitLab

Cloud data environments

Tools

GitHub

GitLab

Databricks Asset Bundles

REST APIs

Job description

Summary:

We are looking for a seasoned Senior Data Engineer with strong Databricks expertise to contribute to the expansion of our enterprise Databricks data platform. The selected candidate will focus on integrating additional datasets, engineering scalable data pipelines, and building new business dimensions and measures within the existing Databricks ecosystem.

Key Responsibilities:

• Architect and build scalable data pipelines leveraging Databricks and PySpark
• Connect and incorporate additional data sources into the existing Databricks Lakehouse environment
• Create and refine dimensional models, business dimensions, measures, and curated datasets for analytical use
• Prepare, transform, and optimize data to support reporting, analytics, and downstream applications
• Contribute to data quality initiatives, validation activities, monitoring, and performance tuning
• Investigate and resolve platform-related issues throughout the project.

Platform & Data Management

• Apply Databricks development, deployment, security, and governance best practices across solutions
• Use Unity Catalog to address data governance and access control requirements
• Build, package, and deploy solutions through Databricks Asset Bundles
• Work closely with the project team to convert business and technical requirements into scalable data solutions.

DevOps & Automation

• Establish and manage CI/CD workflows for data engineering components
• Use GitHub and/or GitLab to manage source code, branching approaches, and peer code reviews
• Contribute to automated deployment workflows and release management activities.

Required Skills:

• 8+ years of professional experience in Data Engineering
• 4+ years of practical, hands-on experience with Databricks (required)
• Strong proficiency with:
• Databricks
• PySpark
• Python
• SQL
• Unity Catalog
• Databricks Asset Bundles
• Databricks Security & Access Management
• Proven experience designing and supporting enterprise-level ETL/ELT solutions
• Extensive experience developing dimensional models, business dimensions, measures, KPIs, and analytical datasets
• Hands-on experience developing and maintaining CI/CD pipelines
• Strong background in source code management using GitHub and/or GitLab
• Experience delivering solutions within cloud-based data environments
• Strong verbal and written communication abilities.

Preferred Skills:

• Exposure to Fivetran or comparable cloud-based data ingestion solutions
• Working knowledge of REST APIs and integrations with enterprise applications
• Experience with Power BI or comparable enterprise business intelligence and reporting platforms
• Background supporting AI, machine learning, or advanced analytics projects
• Experience working with HR, Finance, or other enterprise business data domains.