Databricks -Technical Specialist-Data Engg

Birlasoft; Dadri; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Birlasoft is seeking an experienced AI/GenAI specialist to design and implement enterprise AI solutions using Databricks Genie, Mosaic AI, and MLflow. You will build scalable data pipelines with PySpark and SQL, implement Delta Lake and Unity Catalog, and optimize Spark jobs for performance and cost.

The role includes creating data models, ontologies, and guardrails for GenAI initiatives within an enterprise context.

Qualifications

• Experience designing GenAI solutions with Databricks and building LLM-powered use cases.
• Ability to design scalable data pipelines using PySpark and SQL.
• Hands-on experience implementing Delta Lake, Unity Catalog, and medallion architecture.
• Optimization of Spark jobs for performance, cost, and scalability.
• Ingestion, transformation, and enrichment of data from multiple sources; ensure data quality and observability.

Responsibilities

• Design and implement GenAI and AI/ML solutions using Databricks Genie, Mosaic AI, and related tools.
• Build and deploy LLM-powered use cases and datasets.
• Develop scalable data pipelines using PySpark, SQL; implement Delta Lake and data governance.
• Optimize Spark jobs for performance, cost, and scalability across environments.
• Perform data ingestion and enrichment from multiple sources; ensure data quality and observability.
• Develop enterprise ontologies, taxonomies, vocabularies, and semantic data models; support AI workloads.
• Develop and fine-tune LLMs/ML models using Databricks MLflow and Mosaic AI; implement prompt engineering and guardrails.
• Integrate AI solutions with enterprise data and business workflows.

Skills

GenAI

Databricks Genie

Databricks Agent Bricks

Mosaic AI

PySpark

SQL

Delta Lake

Unity Catalog

Medallion Architecture

Spark Optimization

Data Pipelines

Data Ingestion

Data Transformation

Data Quality

Observability

Ontologies

Taxonomies

Semantic Models

MLflow

Model Fine-tuning

Prompt Engineering

Guardrails

Enterprise GenAI

Data Integration

Tools

Databricks

Python

Spark

Job description

Job Description:

Area(s) of responsibility

Key Responsibilities

• Design and implement GenAI and Agentic AI solutions using Databricks Genie, Databricks Agent Bricks, Mosaic AI (model training, evaluation, prompt engineering, RAG)


• Experience Build LLM-powered use cases


• Design and build scalable data pipelines using Databricks, PySpark, and SQL


• Implement Delta Lake, Unity Catalog, and medallion architecture


• Optimize Spark jobs for performance, cost, and scalability


• Perform data ingestion, transformation, and enrichment from multiple sources


• Ensure data quality, reliability, and observability across pipelines


• Design and develop enterprise ontologies, taxonomies, vocabularies, and semantic data models.


• Support AI/ML workloads by preparing high‑quality, feature‑ready datasets


• Develop and fine-tune LLMs/ML models using Databricks MLflow and Mosaic AI


• Implement prompt engineering, model evaluation, and guardrails for enterprise GenAI use cases


• Integrate AI solutions with enterprise data and business workflows