Data Engineer

First Meridian Business Services; Bengaluru; Hybrid; INR 1,500,000 - 3,000,000; Full time

Job summary

First Meridian Business Services is seeking an expert-level data engineer to design and optimize large-scale data pipelines using Databricks and Delta Lake. You will work with Azure cloud data stack, Spark v3/4, Scala, and related tools to build robust data solutions and scalable workflows.

You will also optimize performance, manage CI/CD using Github Actions/Jenkins, and collaborate on data governance and reporting with Tableau.

Qualifications

• Strong experience with Databricks and building data pipelines.
• Hands-on with Azure cloud data stack.
• Expertise in Spark v3/4, Scala, and Spark ecosystem tools.
• Proficient in performance optimisation with Spark.
• Proficient with Github and usage in CI/CD workflows.
• Understanding of OOP, data structures, SOLID, and clean code.
• Experience with containers and orchestration (Docker, Kubernetes) and S3.
• Exposure to Tableau or similar reporting tools.

Skills

Databricks

Azure cloud data stack

Apache Spark

Scala

Scalatest

Shell scripting

Kubernetes

Airflow

S3

HIVE

Apache Iceberg

Github

OOP

Data structures

SOLID principles

Clean Code

Data Design Patterns

Docker

Tableau

Tools

Nexus

ACR

SonarQube

Snyk

Splunk

New Relic

Github Actions

Jenkins

Job description

Expert-level:

• Strong experience with Databricks(Delta Lake, notebooks, Jobs/ Lakeflow, clusters) & building data pipelines using that
• Hands-on experience with Azure cloud data stack
• Strong expertise on Apache Spark v3/4, Scala, Scalatest, Shell scripting, K8s, Airflow, S3, HIVE, Apache Icebergecosystem
• Proficient in performance optimisation with Spark
• Proficient in source control likeGithub & its usage
• Have proficient understanding of OOP, Data structures, SOLID principles, Clean Code, Data Design Patterns & their application in Data Engineering
• Experience working with containers & container orchestration like Docker, Kubernetesand distributed object storage like S3

Intermediate level:

• Experience on any other distributed SQL engines like Dremio SQL
• Code quality, code security, application monitoring leveraging tools like SonarQube, Snyk, Splunk, New Relic
• Able to work with CICD pipelines using Github Actions, Jenkins
• Artefact repository like Nexus, ACR
• Any reporting tool preferably Tableau
• Data governance using Unity Catalog

Machine Learning and AI

• Candidate should demonstrate ability to work on Machine Learning problems

Candidate should have basic understanding of AI (RAGs/LLMs) & be able to make use of AI related tools & build agents