Data Engineer

First Meridian Business Services; Ahmedabad District; Hybrid; INR 3,500,000 - 5,500,000; Full time

Job summary

First Meridian Business Services is seeking a highly motivated Azure Databricks Engineer with strong expertise in Azure Databricks and cloud-based data platforms. In this role, you will design, develop, and optimize scalable data pipelines, support enterprise-grade data and analytics solutions, and drive best practices in a Databricks environment.

You will collaborate with data engineers, architects, business stakeholders, and analytics teams to enable data-driven decision-making across the

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.

Responsibilities

• Design, develop, and optimize data pipelines using Azure Databricks (Apache Spark).
• Build and support large-scale, cloud-native data processing solutions on Azure.
• Collaborate with data scientists, analysts, and business stakeholders to ensure data availability, reliability, and quality.
• Implement and support Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and MLflow solutions.
• Follow and promote best practices for CI/CD, version control, testing, and deployment in a Databricks environment.
• Work closely with project teams to deliver solutions aligned with business requirements and timelines.

Skills

Azure Databricks

Apache Spark

Python

SQL

Scala

Collibra

Delta Lake

MLflow

Airflow

Cloud computing AWS

CI/CD

Version control

Data warehousing

ETL/ELT

Data modeling

Metadata management

Java

Education

Bachelor's or Master's degree in Computer Science, Data Engineering, or related field

Tools

Azure Databricks

Apache Spark

Delta Lake

Unity Catalog

MLflow

Collibra

Airflow (MWAA)

Job description

Job Summary:

We are seeking a highly motivated and experienced Azure Databricks Engineer with strong expertise in Azure Databricks and cloud-based data platforms. In this role, you will design, develop, and optimize scalable data pipelines, support the implementation of enterprise-grade data and analytics solutions, and leverage Databricks best practices to deliver reliable, high-quality data products. You will work closely with data engineers, architects, business stakeholders, and analytics teams to enable data-driven decision-making across the organization.

Key Responsibilities:

• Design, develop, and optimize data pipelines using Azure Databricks (Apache Spark).
• Build and support large-scale, cloud-native data processing solutions on Azure.
• Collaborate with data scientists, analysts, and business stakeholders to ensure data availability, reliability, and quality.
• Implement and support Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and MLflow solutions.
• Follow and promote best practices for CI/CD, version control, testing, and deployment in a Databricks environment.
• Work closely with project teams to deliver solutions aligned with business requirements and timelines.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• 7+ years of experience in data engineering.
• Hands-on expertise with Azure Databricks, including Spark SQL, Delta Lake, job orchestration, and performance tuning.
• Proficient in Python, SQL, and Apache Spark; knowledge of Scala is a plus.
• Experience working with Azure cloud services, including Databricks deployments.
• Strong understanding of data modeling, ETL/ELT, data warehousing, and modern data lake architectures.
• Advanced SQL skills for data analysis. Expertise on problem-solving skills with a focus on finding innovative solutions to complex data challenges. Ability to proactively identify and address potential issues and bottlenecks.
• Fluent in multiple coding languages, such as Python or Java and able to learn new ones quickly
• Technical skills in metadata capturing and management using Collibra to document, organize, and ensure the quality of metadata assets
• Strong Knowledge of cloud-based data, compute, and storage services, including AWS S3, EC2, RDS, EBS, Lambda, Airflow (MWAA), Containerization Services (ECS, EKS) and so on.
  

Preferred Qualifications:

• Databricks certification (e.g., Databricks Certified Data Engineer Professional).
• Experience with Unity Catalog, MLflow, and building data products on the Lakehouse.
• Experience with Azure services such as Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS).