Databricks Technical Lead

NRM Analytix; Tamil Nadu; On-site; INR 3,000,000 - 5,000,000; Full time

Job summary

NRM Analytix is seeking an experienced Databricks Tech Lead to design, develop, and implement scalable data engineering solutions on the Databricks platform in Tamil Nadu, India.

You will lead a technical team, guide stakeholders, and drive best practices for Delta Lake, Unity Catalog, and ADLS integration with Azure data services, while optimizing Spark/PySpark workloads and building robust data pipelines.

Qualifications

• 7+ years of experience in Data Engineering.
• Strong hands-on Databricks experience.
• Proficient in Apache Spark / PySpark.
• Advanced SQL skills and Delta Lake knowledge.
• Experience with Azure Data Factory, ADLS, Unity Catalog.
• Understanding of Lakehouse architecture and CI/CD pipelines.
• Git version control and collaboration practices.

Responsibilities

• Lead design and development of scalable Databricks-based data engineering solutions.
• Provide technical leadership and conduct design/code reviews.
• Develop and optimize data pipelines with PySpark, Spark SQL, SQL, and Delta Lake.
• Design and implement ETL/ELT workflows using Azure Databricks and ADF.
• Collaborate with architects, data engineers, and stakeholders; troubleshoot issues.
• Establish standards, best practices, and reusable frameworks.
• Support CI/CD and deployment processes; mentor team members.

Skills

Databricks

Apache Spark / PySpark

SQL

Delta Lake

Azure Data Factory

ADLS

Unity Catalog

CI/CD pipelines

Git

Data modeling

Data warehousing

Job description

We are looking for an experienced Databricks Tech Lead to lead the design, development, and implementation of scalable data engineering solutions using the Databricks platform. The ideal candidate should have strong hands-on experience with Azure Databricks, Spark/PySpark, SQL, Delta Lake, Azure Data Factory, and cloud data platforms, along with the ability to guide a technical team and work closely with stakeholders.

Key Responsibilities

• Lead the design and development of scalable Databricks-based data engineering solutions.
• Provide technical leadership to data engineers and conduct design/code reviews.
• Develop and optimize data pipelines using PySpark, Spark SQL, SQL, and Delta Lake.
• Design and implement ETL/ELT workflows using Azure Databricks and Azure Data Factory.
• Work with Delta Lake, Unity Catalog, ADLS, Azure Synapse, and other Azure data services.
• Implement Medallion Architecture including Bronze, Silver, and Gold layers.
• Optimize Spark jobs, Databricks workloads, and SQL queries for performance and cost.
• Design reliable data ingestion and transformation frameworks.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Collaborate with architects, data engineers, business stakeholders, and project teams.
• Troubleshoot complex production and data pipeline issues.
• Establish technical standards, best practices, and reusable frameworks.
• Support CI/CD and deployment processes for Databricks workloads.
• Mentor team members and contribute to technical documentation.

Required Skills

• 7+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Strong knowledge of Apache Spark / PySpark.
• Advanced SQL skills.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of Unity Catalog and Databricks security/governance.
• Strong understanding of data modeling and data warehousing concepts.
• Experience with Git and CI/CD pipelines.
• Strong debugging, problem-solving, and technical leadership skills.

Good to Have

• Databricks certification.
• Experience with Azure Synapse / Microsoft Fabric.
• Experience with Terraform or infrastructure-as-code.
• Experience with streaming technologies such as Kafka / Event Hubs.
• Knowledge of Delta Live Tables / Lakeflow.
• Experience with data governance and security.
• Experience working in large-scale enterprise data environments