Lead Data Engineer, AI

Jobtailor; Bengaluru; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Jobtailor is seeking an experienced software engineer to modify source code for cloud-based architectures with strong Databricks integration and authentication. The role involves building Databricks ingestion pipelines, configuring Azure resources, and creating secure CI/CD workflows.

You will collaborate with stakeholders to test, document, and deliver robust cloud solutions. Strong expertise in FastAPI/React applications,Claude models, and API-driven notifications is required, along with

Qualifications

• Experience modifying source code for cloud-based architectures.
• Proficiency with Databricks data ingestion pipelines.
• Experience with security remediation and testing.
• Strong documentation for cloud architecture and security approvals.
• Ability to work independently in short sprints.

Responsibilities

• Modify source code for architectural requirements, including authentication and Databricks integration.
• Participate in technical reviews for Azure resource configuration and deployment.
• Prepare technical and network architecture documentation for cloud and security approvals.
• Design and build Databricks ingestion pipelines from SAP and website sources.
• Perform scenario-based and edge-case testing and finalize code changes with stakeholders.
• Configure and use Databricks Unity Catalog permissions, OBO tokens, and service principals.
• Build and deploy Databricks Apps using Visual Studio Code.
• Send custom Databricks notifications through APIs to Teams channel webhooks.
• Build minimal FastAPI/React single-page chat apps using Claude models and APIs.

Skills

Databricks Integration

Azure Resource Configuration

FastAPI/React Application Development

CI/CD Pipeline Automation

Security Remediation

Education

Bachelor's degree in a relevant field

Tools

Databricks

Azure DevOps

Visual Studio Code

Claude Models

API Integration

Job description

• Modify source code for architectural requirements, including authentication and Databricks integration
• Participate in technical reviews for Azure resource configuration and deployment
• Prepare technical and network architecture documentation for cloud and security approvals
• Design and build Databricks ingestion pipelines from sources such as SAP and website scraping
• Perform scenario-based and edge-case testing and finalize code changes and acceptance criteria with stakeholders
• Configure and use Databricks Unity Catalog permissions, OBO tokens, and service principals
• Build and deploy Databricks Apps using Visual Studio Code
• Send custom Databricks notifications through APIs to Teams channel webhooks
• Build minimal FastAPI/React single-page chat applications using Claude models and APIs
• Apply prompt engineering, token economics, prompt rule enforcement, AI safety guardrails, and caching mechanisms
• Reengineer code, map code behavior to user requirements, and use graphs to identify dependencies and circular references
• Implement Claude Vision API, NLP processing, and retrieval-augmented generation for Databricks
• Work with Claude Code and Genie
• Troubleshoot full-stack frontend, backend, middleware, data-layer, and routing issues
• Remediate security findings and document code using TDD, FDD, DFD, and SDs
• Create and automate Azure DevOps CI/CD pipelines for Teams bots, web applications, and Databricks deployments
• Add Snyk scans and linting validation steps and configure deployment environments
• Collaborate with client toxicologists and project managers on short requirements
• Deliver work independently in short sprint cycles

Core Competencies

Demonstrates expertise in Databricks integration, cloud architecture documentation, and building data ingestion pipelines. Proficient in implementing security measures and developing applications using FastAPI and React.

Highest-signal resume keywords

• Databricks Integration
• Azure Resource Configuration
• FastAPI/React Application Development
• CI/CD Pipeline Automation
• Security Remediation

Hard Skills

• Source Code Modification
• Databricks Ingestion Pipelines
• Scenario-Based Testing
• Prompt Engineering
• NLP Processing
• Full-Stack Troubleshooting
• TDD
• FDD
• DFD
• SDs

Soft Skills

• Collaboration
• Independent Work
• Stakeholder Communication

Industry Keywords

• Cloud Architecture
• Security Approvals
• Data Layer
• Service Principals
• Teams Bots

Tools & Technologies

• Databricks
• Azure DevOps
• Visual Studio Code
• Claude Models
• API Integration