Databricks Engineer

Grid; Bengaluru; On-site; INR 900,000 - 1,800,000; Full time

Job summary

Grid is seeking a Databricks Engineer to design, develop, and optimise scalable data pipelines and data processing solutions. You will work with Databricks, Apache Spark, PySpark, SQL, and cloud platforms to transform complex datasets into analytics-ready data.

Responsibilities include building ETL/ELT pipelines, optimising Spark jobs, integrating multi-source data, and implementing data governance and security practices. Collaboration with data teams is essential.

Qualifications

• 3–5 years of experience in Data Engineering or a related role.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.

Skills

Databricks

Apache Spark

Python

SQL

Cloud platforms

Data pipelines

ETL/ELT

Tools

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Lakehouse

Git

CI/CD

Workflow orchestration

Azure

AWS

GCP

Job description

We are looking for a skilled Databricks Engineer to design, develop, and optimise scalable data pipelines and data processing solutions. The ideal candidate should have strong experience with Databricks, Apache Spark, Python, SQL, and cloud-based data platforms, with the ability to transform complex datasets into reliable, analytics-ready data.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Build and optimise data processing workflows using PySpark and SQL.
• Work with large datasets to support analytics, reporting, and data-driven applications.
• Develop and manage data lakehouse solutions using Delta Lake and Lakehouse architecture.
• Implement data ingestion, transformation, validation, and quality checks.
• Optimise Spark jobs and Databricks workloads for performance and cost efficiency.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Work with cloud platforms such as Azure, AWS, or Google Cloud.
• Implement workflow orchestration, monitoring, error handling, and troubleshooting.
• Collaborate with data engineers, analysts, and software teams to deliver reliable data solutions.
• Follow best practices for data security, governance, access control, and documentation.

Requirements:

• 3-5 years of experience in Data Engineering or a related role.
• Hands-on experience with Databricks and Apache Spark.
• Strong programming skills in Python and PySpark.
• Strong SQL skills, including query optimisation.
• Experience building and maintaining ETL/ELT data pipelines.
• Knowledge of Delta Lake, data lakehouse architecture, and distributed data processing.
• Experience with at least one cloud platform: Azure, AWS, or GCP.
• Understanding of data modelling, data warehousing, and data quality management.
• Experience with Git, CI/CD, and workflow orchestration tools.

Good to Have:

• Experience working in the utility, energy, power, water, or gas industry.
• Familiarity with utility data, IoT, time-series data, or operational technology systems.
• Experience integrating data from multiple enterprise systems and APIs.
• Knowledge of streaming technologies such as Kafka or Spark Structured Streaming.
• Experience with Unity Catalogue, data governance, and access management.
• Familiarity with BI and visualisation tools such as Power BI or Tableau.