Databricks Tech Lead

Fa Ewjt Saasfaprod1

Bengaluru

Presencial

INR 2.500.000 - 4.500.000

Jornada completa

14 días+

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

EXL Service is seeking a Databricks Platform Engineer to design, build, and sustain Databricks workspaces, clusters, and compute pools across development and production environments. You will configure Unity Catalog for governance, optimize costs, and implement best practices for secrets and access controls.

You will also design Delta Lake architectures, implement Delta Live Tables pipelines, and manage data lakehouse patterns with Delta Lake partitioning, time travel, and CDF.

Formación

  • Proficient in building and maintaining Databricks platforms across environments.
  • Experience with Delta Lake, Delta Live Tables, and data lakehouse patterns.
  • Knowledge of Unity Catalog for governance and security policies.

Responsabilidades

  • Design and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod.
  • Configure Unity Catalog for access control and data lineage.
  • Build Delta Lake tables with partitioning and optimization techniques.
  • Develop PySpark/SQL pipelines for batch and streaming data.
  • Support ML workloads with MLflow tracking and model serving.

Conocimientos

PySpark
SQL
Delta Lake
Unity Catalog
Data governance
MLflow
AI/GenAI workloads
Cloud integration
DevOps practices

Herramientas

Azure DevOps
GitHub Actions
Terraform

Descripción del empleo

Databricks Platform Engineering
  • Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.
  • Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.
  • Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance.
  • Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).
  • Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.
Delta Lake & Lakehouse Architecture
  • Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).
  • Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.
  • Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.
  • Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.
  • Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).

EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com

Data Pipeline Development (PySpark / SQL)
  • Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake.
  • Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables.
  • Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning.
  • Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity.
  • Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines.
MLflow & AI/ML Workloads
  • Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks.
  • Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances.
  • Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features.
  • Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI).
  • Implement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints.
Cloud Integration & DevOps
  • Integrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS).
  • Build and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI.
  • Implement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources.
  • Manage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte).
  • Monitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools.
Governance, Security & Optimization
  • Implement row-level security, column masking, and dynamic data views using Unity Catalog policies.
  • Ensure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations.
  • Conduct performance tuning — query plan analysis, caching strategies, Photon engine enablement.
  • Maintain data cataloging, metadata management, and data lineage tracking within Unity Catalog.
  • Document architecture decisions, runbooks, and operational guides for Databricks workloads.
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Databricks Engineer
Databricks Engineer

Fa Ewjt Saasfaprod1 • Chennai

Presencial
INR 3.500.000 - 6.000.000
Data Architect I
Data Architect I

UST • Pune District

Presencial
INR 4.000.000 - 7.000.000
Senior Manager
Senior Manager

ExlService Holdings, Inc. • Bengaluru

Híbrido
INR 2.400.000 - 3.600.000
Data Engineer (Databricks ) Senior Manager
Data Engineer (Databricks ) Senior Manager

PwC • Pune District, Gurugram District, Bengaluru

Híbrido
INR 3.500.000 - 5.500.000
Technical Lead
Technical Lead

Carelon Global Solutions • Gurugram District, Bengaluru

Híbrido
INR 3.000.000 - 5.500.000
Databricks Engineer
Databricks Engineer

Infosys • Bengaluru

Presencial
INR 2.800.000 - 4.400.000
Cloudera Data Engineer
Cloudera Data Engineer

McLaren Strategic Ventures • Bengaluru

Presencial
INR 6.000.000 - 9.000.000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Bestal • Hyderabad

Híbrido
INR 4.000.000 - 7.000.000
Senior Data Engineer
Senior Data Engineer

Bridgestone Americas, Inc. • Bengaluru Urban

Presencial
INR 1.500.000 - 2.300.000
Azure Databricks Developer
Azure Databricks Developer

Birlasoft • Pune District

Presencial
INR 900.000 - 1.500.000