Databricks Developer

Tekskills Inc.; Kolkata District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tekskills Inc., Kolkata, is seeking a Data Engineer to architect scalable data pipelines using Databricks, Spark and Delta Lake. You will implement the Bronze-Silver-Gold medallion architecture, design data models, and ensure quality and security across the pipeline.

Collaborate with data scientists and business stakeholders to deliver analytics-ready data. The ideal candidate has Databricks Certified Associate Developer for Apache Spark (or equivalent), strong Python/Scala skills, and

Qualifications

• Databricks Certified Associate Developer for Apache Spark (or equivalent).
• Strong Python/Scala for Spark development.
• Experience with Delta Lake and Spark DataFrame API.
• Proven experience migrating from Cloudera to Databricks.
• Data pipeline orchestration (Databricks Jobs, Airflow, etc.).
• Data quality and security best practices.

Responsibilities

• Data Pipeline Design: Architect and implement scalable pipelines using Databricks, Spark and Delta Lake.
• Medallion Architecture Implementation: Bronze to Silver to Gold data flow for analytics.
• Data Modeling: Design and optimize data models for storage and analytics.
• Data Quality & Security: Implement validation, quality checks and security controls.
• Automation & Monitoring: Automate pipeline execution and monitor performance.
• Collaboration: Work with data scientists, analysts and stakeholders to deliver analytics-ready data.

Skills

Databricks

Python/Scala for Spark

Delta Lake / Spark DataFrame API

Cloudera migration to Databricks

Data pipeline orchestration (Databrics

Tools

Databricks

Spark

Delta Lake

Airflow

Docker

Kubernetes

Job description

Job Description

• Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake
• Medallion Architecture Implementation: Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer
• Data Modeling: Design and optimize data models for efficient storage, retrieval, and analytics
• Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline
• Automation & Monitoring: Automate pipeline execution, monitor performance, and troubleshoot issues
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality, analytics-ready data

Essential Skills

• Databricks Certified Associate Developer for Apache Spark (or equivalent)
• Strong Python/Scala for Spark development
• Experience with Delta Lake and Spark DataFrame API
• Proven experience migrating from Cloudera to Databricks
• Data pipeline orchestration (Databricks Jobs, Airflow, etc.)
• Data quality and security best practices

Desirable Skills

• Databricks Certified Data Engineer Associate
• SQL and advanced query optimization
• Cloud platforms (Azure, AWS, GCP)
• Real-time/streaming data processing (Spark Structured Streaming)
• DevOps/MLOps (CI/CD, Docker, Kubernetes)
• Experience with data governance and lineage tools