Databricks Technical Lead

Cai Info India; Bengaluru; Hybrid; INR 4,000,000 - 5,500,000; Full time

Job summary

CAI seeks a Databricks Technical Lead to shape a strategy for AI-ready data products, ensuring governance, quality, and scalable pipelines across enterprise data sources. The role is full-time and hybrid in Bengaluru, with responsibilities spanning roadmap ownership, data contracts, and cost optimization.

You will collaborate with analytics, AI delivery, and security teams to mature data products from concept to production, enabling reliable AI solutions while controlling costs and maintaining

Qualifications

• Bachelor’s degree or higher in Computer Science, Data Science, AI/ML, Applied Mathematics, Engineering, or related field
• 8+ years of experience in data engineering, data management, AI data foundations, or related technology leadership roles
• Demonstrated ownership of data products or mission-critical data pipelines delivering measurable business impact
• Hands-on experience with AWS, Databricks, Delta Lake, Unity Catalog, data lineage, observability, and production support
• Experience defining and managing data contracts, quality gates, service levels, and data access models
• Strong stakeholder management skills with experience partnering across business leaders, product owners, data and AI engineering teams, security, and platform organizations
• Ability to operate at a senior level, challenge incomplete requirements, evaluate trade-offs, and drive decisions in ambiguous environments
• Strong communication skills with the ability to simplify complex technical concepts for business stakeholders
• Proven ownership mindset with accountability for business and technical outcomes
• Strong judgment balancing speed, quality, security, cost, and maintainability
• Excellent problem-solving skills and ability to adapt in fast-paced, evolving environments
• Commitment to data quality, operational excellence, documentation, and continuous improvement

Responsibilities

• Own the AI data product roadmap and translate business priorities into a sequenced backlog of reusable data products with defined ownership, service levels, and lifecycle controls
• Partner with dashboard, analytics, and AI delivery teams to identify reporting assets that should evolve into governed AI-ready data products
• Establish data product charters defining purpose, consumers, source ownership, data contracts, access models, and retention requirements
• Implement table-, column-, and row-level access standards, classification, audit trails, and data handling controls using Databricks Unity Catalog and AWS services
• Lead the design and operation of incremental, observable, and cost-efficient data ingestion and curation pipelines across enterprise data sources
• Create and maintain versioned, reproducible training datasets, evaluation datasets, inference inputs, feature tables, and monitoring datasets with clear metadata and documentation
• Define and implement AI-specific data quality standards including completeness, consistency, timeliness, uniqueness, referential integrity, distribution stability, and drift sensitivity
• Establish automated quality controls and enforce readiness gates that prevent model training, deployment, or production promotion when critical data quality requirements are not met
• Develop operational runbooks for recovery, backfills, reprocessing, incident response, and escalation management
• Participate in operational reviews and drive root-cause analysis and corrective actions to improve platform reliability
• Optimize storage, compute utilization, partitioning, scheduling, and overall platform cost efficiency
• Approve data readiness gates for model training, evaluation, deployment, and significant production inference input changes
• Escalate data ownership, quality, access, and cost issues that introduce risk or impede business outcomes
• Drive dashboard-to-data-product transformations and establish governed, reusable data products that support critical AI use cases
• Deliver quality and cost scorecards that provide transparency into data health, reliability, usage, and platform cost drivers

Skills

Data engineering

Data management

AI data foundations

Leadership

Stakeholder management

Education

Bachelor’s degree or higher in Computer Science, Data Science, AI/ML or related field

Tools

AWS

Databricks

Delta Lake

Unity Catalog

Data lineage

Job description

Databricks Technical Lead

Req number: R8395

Employment type: Full time

Worksite flexibility: Hybrid

Who we are

CAI is a global services firm with over 9,000 associates worldwide and a yearly revenue of $1.3 billion+. We have over 40 years of excellence in uniting talent and technology to power the possible for our clients, colleagues, and communities. As a privately held company, we have the freedom and focus to do what is right—whatever it takes. Our tailor-made solutions create lasting results across the public and commercial sectors, and we are trailblazers in bringing neurodiversity to the enterprise.

Job Summary

We are looking for a motivated Databricks Technical Lead ready to take us to the next level! If you have experience with AWS, Databricks, and AI data product management, and are looking for your next career move.

Job Description

We are looking for a Databricks Technical Lead to lead the strategy, governance, productization, and operational excellence of AI-ready data products that enable scalable and reliable AI solutions across the enterprise. This position will be full-time and hybrid in Bengaluru.

What You’ll Do

• Own the AI data product roadmap and translate business priorities into a sequenced backlog of reusable data products with defined ownership, service levels, and lifecycle controls

• Partner with dashboard, analytics, and AI delivery teams to identify reporting assets that should evolve into governed AI-ready data products

• Establish data product charters defining purpose, consumers, source ownership, data contracts, access models, and retention requirements

• Implement table-, column-, and row-level access standards, classification, audit trails, and data handling controls using Databricks Unity Catalog and AWS services

• Lead the design and operation of incremental, observable, and cost-efficient data ingestion and curation pipelines across enterprise data sources

• Create and maintain versioned, reproducible training datasets, evaluation datasets, inference inputs, feature tables, and monitoring datasets with clear metadata and documentation

• Define and implement AI-specific data quality standards including completeness, consistency, timeliness, uniqueness, referential integrity, distribution stability, and drift sensitivity

• Establish automated quality controls and enforce readiness gates that prevent model training, deployment, or production promotion when critical data quality requirements are not met

• Develop operational runbooks for recovery, backfills, reprocessing, incident response, and escalation management

• Participate in operational reviews and drive root-cause analysis and corrective actions to improve platform reliability

• Optimize storage, compute utilization, partitioning, scheduling, and overall platform cost efficiency

• Approve data readiness gates for model training, evaluation, deployment, and significant production inference input changes

• Escalate data ownership, quality, access, and cost issues that introduce risk or impede business outcomes

• Drive dashboard-to-data-product transformations and establish governed, reusable data products that support critical AI use cases

• Deliver quality and cost scorecards that provide transparency into data health, reliability, usage, and platform cost drivers

What You’ll Need

Required:

• Bachelor’s degree or higher in Computer Science, Data Science, AI/ML, Applied Mathematics, Engineering, or a related field

• 8+ years of experience in data engineering, data management, AI data foundations, or related technology leadership roles

• Demonstrated ownership of data products or mission-critical data pipelines delivering measurable business impact

• Hands-on experience with AWS, Databricks, Delta Lake, Unity Catalog, data lineage, observability, and production support

• Experience defining and managing data contracts, quality gates, service levels, and data access models

• Strong stakeholder management skills with experience partnering across business leaders, product owners, data and AI engineering teams, security, and platform organizations

• Ability to operate at a senior level, challenge incomplete requirements, evaluate trade-offs, and drive decisions in ambiguous environments

• Strong communication skills with the ability to simplify complex technical concepts for business stakeholders

• Proven ownership mindset with accountability for business and technical outcomes

• Strong judgment balancing speed, quality, security, cost, and maintainability

• Excellent problem-solving skills and ability to adapt in fast‑paced, evolving environments

• Commitment to data quality, operational excellence, documentation, and continuous improvement

Preferred:

• Databricks Data Engineer Professional certification

• Databricks Machine Learning Professional certification

• AWS Certified Data Analytics certification

• AWS Solutions Architect Associate certification

• AWS Solutions Architect Professional certification

• Data governance, data management, product ownership, or agile delivery certifications

Physical Demands

• Ability to safely and successfully perform the essential job functions

• Sedentary work that involves sitting or remaining stationary most of the time with occasional need to move around the office to attend meetings, etc.

• Ability to conduct repetitive tasks on a computer, utilizing a mouse, keyboard, and monitor

Reasonable accommodation statement

If you require a reasonable accommodation in completing this application, interviewing, completing any pre‑employment testing, or otherwise participating in the employment selection process, please direct your inquiries to [e-mail] or (888) 824 – 8111.