Databricks Developer

Radiant Systems; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Radiant Systems in Bengaluru seeks a Databricks Developer to design, implement, and optimize scalable data processing using Apache Spark, PySpark, and Delta Lake on the Databricks platform. You will ensure reliable batch and CDC pipelines, cloud integration with AWS services, and performance-tuned storage and processing.

The role emphasizes modular development, version control, testing, and automation, with strong focus on data quality, reliability, and cost optimization across Databricks

Qualifications

• Experience with Spark-based data processing at scale.
• Proficiency in Python/PySpark development.
• Hands-on Delta Lake and Delta table management.
• Experience building Databricks notebooks, jobs and workflows.
• AWS cloud integration with S3, EC2, ECS, DynamoDB.

Responsibilities

• Design, build, and maintain Spark/PySpark data pipelines.
• Develop Delta Lake solutions with quality controls.
• Implement batch and CDC pipelines for incremental processing.
• Optimize Spark jobs, storage layouts, and pipeline execution.

Skills

Apache Spark

Python (PySpark)

Delta Lake

Databricks

AWS S3

CDC pipelines

Spark optimization

REST APIs

Tools

Databricks

Delta Lake

Job description

We are hiring for Databricks Developer in Bangalore location

Role Summary

The Databricks Developer will design, develop, test, and optimize scalable data-processing solutions using Apache Spark, PySpark, Delta Lake, and the Databricks platform. The role requires practical experience in reliable batch and change-data-capture pipelines, cloud integration, workload troubleshooting, and storage and processing optimization.

Key Responsibilities

• Design, build, and maintain scalable Spark and PySpark data pipelines.


• Develop Delta Lake solutions with effective schema handling, data quality controls, and read/write patterns.


• Implement batch and CDC pipelines for incremental processing.


• Optimize Spark jobs, storage layouts, and pipeline execution for performance, reliability, and cost.


• Develop Databricks notebooks, workflows, jobs, and reusable components.


• Integrate Databricks workloads with AWS services such as S3, EC2, ECS, and DynamoDB.


• Troubleshoot pipeline failures, data-quality issues, performance bottlenecks, and production incidents.


• Apply modular development, version control, testing, code review, deployment automation, and documentation practices.


• Use REST APIs and platform automation where required.

Must-Have Technical Requirements

• Apache Spark knowledge from core concepts through distributed processing patterns.


• Production-grade Python and PySpark development.


• Practical Delta Lake and Delta table experience.


• Storage, file-layout, Spark execution, and pipeline optimization.


• Databricks notebooks, jobs, workflows, compute, and development practices.


• Strong in Python data structures and algorithms.


• AWS data-engineering services, particularly S3, with familiarity with EC2, ECS, and DynamoDB.


• CDC and incremental data-processing solutions.


• REST APIs for integration and automation.


• Ability to develop solutions using Databricks Apps when required.

Good-to-Have Qualifications

• Spark Structured Streaming fundamentals.


• Exposure to Mosaic AI capabilities.


• Relevant Databricks certification or preparation toward certification.

Expected Developer Profile

The preferred candidate is a hands-on developer who can translate requirements into maintainable Databricks solutions. The strongest emphasis is on Spark, PySpark, Delta Lake, pipeline and storage optimization, AWS integration, and CDC