Databricks - Data Engineer

Tredence Inc.; Bengaluru; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Tredence Inc. is seeking a highly skilled Senior Databricks Engineer to join our team in Bengaluru, Karnataka. In this client-facing role, you will design, develop, and maintain scalable enterprise-grade data solutions using Databricks and PySpark.

The ideal candidate should have a minimum of 6 years of hands-on experience in Data Engineering, strong communication skills, and the ability to optimize large-scale data pipelines. Join us to drive innovation and optimize data engineering practices.

Qualifications

• Minimum 6+ years of hands-on experience in Data Engineering using Databricks.
• Strong hands-on experience with Databricks, PySpark, SQL, and PLSQL.
• Experience with CI/CD processes and Pull Request workflows.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build ingestion frameworks for batch and streaming data workloads.
• Optimize Spark jobs and SQL queries for performance.

Skills

Databricks

PySpark

SQL

PLSQL

Data Engineering

Unity Catalog (UC)

Spark Streaming

CI/CD

Data Quality frameworks

Job description

We are looking for a highly skilled and hands-on Senior Databricks Engineer to join our team. This is a client-facing role requiring strong technical expertise, ownership mindset, proactive communication, and the ability to build scalable enterprise-grade data solutions.

The ideal candidate should have deep experience in Databricks, PySpark, SQL/PLSQL, modern data engineering practices, CI/CD processes, and large-scale distributed data processing. The candidate should be comfortable working in fast-paced environments, troubleshooting complex data engineering problems, and driving optimization initiatives.

Key Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines using Databricks and PySpark.
• Build and manage ingestion frameworks for batch and streaming data workloads.
• Develop enterprise-grade data solutions using Delta Lake and Medallion Architecture principles.
• Implement and manage data pipelines using DLT (Delta Live Tables) and Spark Streaming.
• Work extensively with Unity Catalog (UC) for governance and data access management.
• Optimize Spark jobs, SQL queries, and distributed processing workloads for performance and cost efficiency.
• Participate in CI/CD and release management processes using Git and Databricks Asset Bundles.
• Collaborate with business stakeholders, architects, and client teams to understand requirements and deliver robust solutions.
• Troubleshoot production issues, identify root causes, and implement sustainable fixes.
• Drive engineering best practices, code quality standards, and reusable framework development.
• Proactively communicate risks, blockers, and improvement opportunities.
• Challenge existing processes and contribute ideas to improve platform scalability, reliability, and engineering maturity.

Must-Have Skills & Experience

Technical Skills

• Minimum 6+ years of hands-on experience in Data Engineering (using Databricks)
• Strong hands-on experience with:
• Databricks
• PySpark
• SQL
• PLSQL
• Strong experience building large-scale and complex data pipelines.
• Hands-on experience with:
• Medallion Architecture
• Unity Catalog (UC)
• Spark Streaming
• Experience designing and developing ingestion frameworks.
• Strong troubleshooting and performance tuning skills in Spark/Databricks environments.
• Experience with CI/CD processes, Pull Request (PR) workflows, Git, and Databricks Asset Bundles.
• Experience working with distributed data processing and optimization techniques.
• Strong understanding of data engineering best practices and scalable architecture patterns.

Soft Skills

• Excellent commitment and ownership mindset.
• Strong client-facing communication and stakeholder management skills.
• Proactive communicator with the ability to challenge status quo constructively.
• Strong analytical and problem-solving abilities.
• Ability to work independently in fast-paced delivery environments.

Nice-to-Have Skills

• Experience implementing Data Quality frameworks and controls.
• Data Modeling skills (dimensional modeling, warehouse modeling, etc.).
• Exposure to cloud-native data platforms (preferably Azure) and enterprise data governance practices.
• Experience working in Agile/Scrum delivery models.