Databricks Developer

Appbaytech; Jaipur; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Appbay Technologies is seeking an experienced Databricks Developer for a 6-month contractual engagement. You will design, develop, and optimize scalable data solutions using Databricks, Spark, PySpark, and Delta Lake.

You will build data pipelines, implement ETL/ELT processes, and collaborate with engineers, analysts, and stakeholders to ensure high data quality, security, and reliability.

Qualifications

• 5–8 years of experience in Data Engineering or Databricks development.
• Strong hands-on Databricks expertise with Spark and PySpark.
• Excellent SQL skills and experience with Delta Lake.
• Solid ETL/ELT design and data pipeline development.
• Experience on Azure/AWS/GCP cloud platforms and data lake architectures.
• Knowledge of data warehousing concepts and CI/CD practices.
• Proficient in Git version control and collaborative development.

Responsibilities

• Develop and maintain scalable data pipelines using Databricks and Spark.
• Design and implement ETL/ELT workflows for large datasets.
• Create data solutions using PySpark and SQL.
• Work with Delta Lake for reliable data storage.
• Optimize Spark jobs and queries for performance and cost.
• Build and manage workflows with Databricks Workflows.
• Integrate data from multiple sources into Databricks.
• Perform data transformation, cleansing, and validation.
• Collaborate with data engineers, analysts, and stakeholders.
• Troubleshoot data pipeline issues and ensure data quality.
• Follow data security, governance, and development best practices.
• Participate in code reviews, testing, and deployment activities.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

ETL/ELT

Cloud Platforms

Data Warehousing

Git & CI/CD

Analytical Skills

Job description

Experience: 5–8 Years
Engagement: Contractual
Contract Duration: 6 Months
Joining: Immediate / Short Notice Preferred

About the Role

Appbay Technologies is looking for an experienced Databricks Developer to join our team on a 6-month contractual engagement. The ideal candidate will have strong hands‑on experience in Databricks, data engineering, Spark, and cloud‑based data platforms and will be responsible for designing, developing, and optimizing scalable data solutions.

Key Responsibilities

• Develop and maintain scalable data pipelines using Databricks and Apache Spark.
• Design and implement ETL/ELT workflows for processing large datasets.
• Develop data solutions using PySpark and/or SQL.
• Work with Delta Lake for reliable and scalable data storage.
• Optimize Spark jobs, queries, and data pipelines for performance and cost efficiency.
• Build and manage workflows using Databricks Workflows.
• Integrate data from multiple sources into Databricks.
• Implement data transformation, cleansing, and validation processes.
• Collaborate with data engineers, analysts, developers, and business stakeholders.
• Troubleshoot data pipeline issues and ensure data quality and reliability.
• Follow best practices for data security, governance, and development.
• Participate in code reviews, documentation, testing, and deployment activities.

Required Skills

• 5–8 years of experience in Data Engineering / Databricks development.
• Strong hands‑on experience with Databricks.
• Strong knowledge of Apache Spark and PySpark.
• Excellent proficiency in SQL.
• Experience with Delta Lake and data lake architecture.
• Strong understanding of ETL/ELT concepts and data pipelines.
• Experience working with cloud platforms such as Azure, AWS, or GCP.
• Good understanding of data warehousing and data modeling concepts.
• Experience with Git/version control and CI/CD practices.
• Strong analytical and problem‑solving skills.

Good to Have

• Experience with Azure Databricks.
• Knowledge of Azure Data Factory / AWS Glue / similar data integration tools.
• Experience with Unity Catalog and Databricks governance.
• Knowledge of cloud storage such as ADLS, S3, or GCS.
• Experience with streaming technologies such as Kafka.
• Databricks certifications will be an added advantage.

Unlock Your Potential with Us!