Databricks Developer

Tekskills Inc.; Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tekskills Inc. in Mumbai is seeking a data pipeline specialist to design and implement scalable data pipelines using Databricks, Spark, and Delta Lake. You will migrate from Cloudera to Databricks and build Bronze-Silver-Gold architectures for analytics.

You will collaborate with data scientists and analysts, ensure data quality and security, and automate workflows with Databricks Jobs and Airflow. Proficiency in Python/Scala and SQL is essential.

Qualifications

• Databricks Certified Associate Developer for Apache Spark (or equivalent) required.
• Strong Python/Scala for Spark development.
• Experience with Delta Lake and Spark DataFrame API.
• Proven experience migrating from Cloudera to Databricks.
• Data pipeline orchestration using Databricks Jobs or Airflow.
• Data quality and security best practices.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Spark.
• Ingest raw data across Bronze/Silver/Gold layers and process for analytics.
• Model data for efficient storage, retrieval, and analytics.
• Enforce data quality, validation, and security controls across pipelines.
• Automate pipeline execution and monitor performance.
• Collaborate with data scientists, analysts, and stakeholders to deliver analytics-ready data.

Skills

Databricks Certified Associate Dev for

Python/Scala for Spark

Delta Lake and Spark DataFrame API

Cloudera to Databricks migration

Data pipeline orchestration

Data quality and security

Tools

Databricks

Airflow

Docker

Kubernetes

Spark

Delta Lake

Job description

Job Description

• Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake
• Medallion Architecture Implementation: Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer
• Data Modeling: Design and optimize data models for efficient storage, retrieval, and analytics
• Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline
• Automation & Monitoring: Automate pipeline execution, monitor performance, and troubleshoot issues
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality, analytics-ready data

Essential Skills

• Databricks Certified Associate Developer for Apache Spark (or equivalent)
• Strong Python/Scala for Spark development
• Experience with Delta Lake and Spark DataFrame API
• Proven experience migrating from Cloudera to Databricks
• Data pipeline orchestration (Databricks Jobs, Airflow, etc.)
• Data quality and security best practices

Desirable Skills

• Databricks Certified Data Engineer Associate
• SQL and advanced query optimization
• Cloud platforms (Azure, AWS, GCP)
• Real-time/streaming data processing (Spark Structured Streaming)
• DevOps/MLOps (CI/CD, Docker, Kubernetes)
• Experience with data governance and lineage tools