Databricks Developer

Tekskills; Chennai District; On-site; INR 900,000 - 1,600,000; Full time

Job summary

Tekskills is seeking a data engineer to design and implement scalable data pipelines using Databricks, Spark, and Delta Lake in Chennai. You will architect Bronze-Silver-Gold layers, ensure data quality and security, and collaborate with data scientists and analysts to deliver analytics-ready data.

The role requires strong Python/Scala and experience migrating from Cloudera to Databricks, with exposure to Airflow, Docker, and Kubernetes in a cloud environment.

Qualifications

• Experience building scalable data pipelines with Databricks, Spark, and Delta Lake.
• Strong Python/Scala development for Spark workloads.
• Proven track record migrating data platforms from Cloudera to Databricks.

Responsibilities

• Design and implement data pipelines across Bronze-Silver-Gold layers.
• Ensure data quality, security, and governance across pipelines.
• Collaborate with data scientists and analysts to deliver analytics-ready data.

Skills

Databricks

Apache Spark

Python/Scala

Delta Lake

Cloudera migration

Data quality

Data security

Airflow

CI/CD

Tools

Databricks Jobs

Airflow

Docker

Kubernetes

CI/CD tooling

Job description

Job Description

• Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake
• Medallion Architecture Implementation: Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer
• Data Modeling: Design and optimize data models for efficient storage, retrieval, and analytics
• Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline
• Automation & Monitoring: Automate pipeline execution, monitor performance, and troubleshoot issues
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality, analytics-ready data

Essential Skills

• Databricks Certified Associate Developer for Apache Spark (or equivalent)
• Strong Python/Scala for Spark development
• Experience with Delta Lake and Spark DataFrame API
• Proven experience migrating from Cloudera to Databricks
• Data pipeline orchestration (Databricks Jobs, Airflow, etc.)
• Data quality and security best practices

Desirable Skills

• Databricks Certified Data Engineer Associate
• SQL and advanced query optimization
• Cloud platforms (Azure, AWS, GCP)
• Real-time/streaming data processing (Spark Structured Streaming)
• DevOps/MLOps (CI/CD, Docker, Kubernetes)
• Experience with data governance and lineage tools