Sr AI Engineer

Enable Data Incorporated; Hyderabad; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Databricks is seeking engineers to design and build data ingestion pipelines into the Databricks platform, integrating SAP and web sources with PySpark and PySQL.

You will implement tests, discuss edge cases with stakeholders, and manage Unity Catalog permissions with service principals for various Databricks components.

Qualifications

• Experience with PySpark and PySQL for data pipelines.
• Familiarity with Databricks Unity Catalog and service principals.
• Knowledge of Databricks Apps and deploying via VS Code.
• API integration and webhook notifications experience.
• Prompt engineering for AI workflows.

Responsibilities

• Design and build data ingestion pipelines into Databricks from SAP and web sources.
• Write tests and discuss edge cases with stakeholders.
• Manage Unity Catalog permissions, OBO tokens and service principals for Agents/Genie Spaces/ML.
• Develop and deploy Databricks Apps using Visual Studio Code.
• Send custom notifications from Databricks via APIs to team channels.
• Use FastAPI/React to build minimal SPAs and chat interfaces with Claude models.

Tools

Claude Code

Genie

Job description

Databricks

1. Design and build pipelines for ingesting data into Databricks from various sources like SAP, website scraping. Core technical skill will be PySpark and PySQL
2. Write and do scenario-based testing, edge cases and discuss with the stake holders to finalize the necessary code changes and acceptance criteria.
3. Understanding of the Databricks Unity Catalog permissions model and how to use OBO tokens and configure Service Principals for Databricks Agents,Genie Spaces, Machine Learning and Foundation Models.
4. Knowledge of Databricks Apps. Ability to build and Deploy Databricks Apps using Visual Studio Code.
5. Sending custom notifications from Databricks using APIs to custom Team Channel Webhooks

AI:

1. Ability to use FastAPI /React front end and build minimal Single Page Applications(SPAs) using Chat UI interfaces and Claude Models/APIs
2. Ability to use defined System Prompts, User Prompts etc..(i..e,) skills in prompt engineering.
3. Basic Knowledge of Token Economics and ability use rule enforcement in prompts in addition to standard NLP language.
4. Ability to reengineer code and see what it does and map it to the requirement specifications from users
5. Ability to use Graphs to detect inter dependencies between rules and circular references.
6. Ability to use both Claude Vision API in addition to standard text processing.
7. Understand and develop Retrieval Augmented Generation specifically to Databricks like AI Search indexes, Hierarchical Chunking etc..
8. Know how to implement basic guard rails related to AI safety
9. Understand various caching mechanisms

Tools:

1. Ability to work with Claude Code and Genie.