Senior Data Engineer

First Meridian Business Services; Bengaluru; Hybrid; INR 900,000 - 1,500,000; Full time

Job summary

First Meridian Business Services in Bengaluru seeks an expert data engineer to design and optimize large-scale data pipelines using Databricks, Delta Lake, and Spark. You will work on Azure cloud stack, Kubernetes, Docker, Airflow, and S3-based data movement.

The role demands strong performance tuning, code quality practices, and hands-on experience with Git-based workflows. Immediate joiners preferred with deep knowledge of data design patterns.

Qualifications

• Strong experience building data pipelines with Databricks and Delta Lake.
• Hands-on Azure cloud data stack expertise and Spark-based workloads.
• Proficiency in Scala, Spark, Scalatest, and shell scripting.
• Kubernetes, Docker, Airflow, and distributed storage (S3) familiarity.
• Code quality, CI/CD with Git workflows, and performance optimisation.

Skills

Data pipeline design

Performance optimization

Code quality

Git workflows

OOP

SOLID principles

Data structures

Clean Code

Tools

Databricks

Azure data stack

Apache Spark

Scala

Scalatest

Shell scripting

Kubernetes

Airflow

S3

Hive

Apache Iceberg

Github

Docker

Nexus / Artifactory

Job description

Expert-level

• Strong experience with Databricks (Delta Lake, notebooks, Jobs/ Lakeflow, clusters) & building data pipelines using that
• Hands-on experience with Azure cloud data stack
• Strong expertise on Apache Spark v3/4, Scala, Scalatest, Shell scripting, K8s, Airflow, S3, HIVE, Apache Iceberg ecosystem
• Proficient in performance optimisation with Spark
• Proficient in source control like Github & its usage
• Have proficient understanding of OOP, Data structures, SOLID principles, Clean Code, Data Design Patterns & their application in Data Engineering
• Experience working with containers & container orchestration like Docker, Kubernetes and distributed object storage like S3

Intermediate level

• Experience on any other distributed SQL engines like Dremio SQL
• Code quality, code security, application monitoring leveraging tools like SonarQube, Snyk, Splunk, New Relic
• Able to work with CICD pipelines using Github Actions, Jenkins
• Artefact repository like Nexus, ACR
• Any reporting tool preferably Tableau
• Data governance using Unity Catalog

Good to Have

• Machine Learning and AI
  • Candidate should demonstrate ability to work on Machine Learning problems

Candidate should have basic understanding of AI (RAGs/LLMs) & be able to make use of AI related tools & build agents

Notice Period - only Immediate Joiner Or Serving ( Who can join in SEP )