Data

First Meridian Business Services; Bengaluru; Hybrid; INR 3,500,000 - 7,000,000; Full time

Job summary

First Meridian Business Services is seeking an expert-level data engineering professional in Bengaluru to design and optimize large data pipelines using Databricks, Spark, and the Azure data stack. The role emphasizes building scalable pipelines, tuning Spark workloads, and implementing robust data models.

You will work with Docker/Kubernetes, GitHub, and CI/CD tools to deliver reliable data solutions while applying SOLID and clean code principles across the data platform.

Qualifications

• Strong experience with Databricks and building data pipelines using that platform.
• Hands-on experience with Azure cloud data stack.
• Strong expertise on Apache Spark v3/4, Scala, Scalatest, Shell scripting, K8s, Airflow, S3, Hive, Apache Iceberg ecosystem.
• Proficient in performance optimisation with Spark.
• Proficient in source control like Github and its usage.
• Understanding of OOP, data structures, SOLID, Clean Code and data design patterns in data engineering.
• Experience with containers and orchestration (Docker, Kubernetes) and object storage like S3.

Skills

Databricks

Azure cloud data stack

Apache Spark

Scala

Scalatest

Shell scripting

Kubernetes

Airflow

S3

Hive

Apache Iceberg

GitHub

OOP & Clean Code

Data design patterns

Docker

Dremio SQL

SonarQube

Snyk

Splunk

New Relic

GitHub Actions

Jenkins

Nexus

ACR

Tableau

Unity Catalog

Machine Learning & AI basics

Build agents

Tools

Databricks

Azure

Spark

Kubernetes

Docker

S3

Tableau

Job description

JD Attached

Expert-level:

• Strong experience with Databricks (Delta Lake, notebooks, Jobs/ Lakeflow, clusters) & building data pipelines using that
• Hands-on experience with Azure cloud data stack
• Strong expertise on Apache Spark v3/4, Scala, Scalatest, Shell scripting, K8s, Airflow, S3, HIVE, Apache Iceberg ecosystem
• Proficient in performance optimisation with Spark
• Proficient in source control like Github & its usage
• Have proficient understanding of OOP, Data structures, SOLID principles, Clean Code, Data Design Patterns & their application in Data Engineering
• Experience working with containers & container orchestration like Docker, Kubernetes and distributed object storage like S3

Intermediate level:

• Experience on any other distributed SQL engines like Dremio SQL
• Code quality, code security, application monitoring leveraging tools like SonarQube, Snyk, Splunk, New Relic
• Able to work with CICD pipelines using Github Actions, Jenkins
• Artefact repository like Nexus, ACR
• Any reporting tool preferably Tableau
• Data governance using Unity Catalog

Machine Learning and AI

• Candidate should demonstrate ability to work on Machine Learning problems

Candidate should have basic understanding of AI (RAGs/LLMs) & be able to make use of AI related tools & build agents