Senior Databricks Engineer / Tech Lead

HypTechie; Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

HypTechie seeks a Certified Senior Databricks Engineer / Tech Lead with 7–8 years of hands-on experience to design, build, and operationalize scalable data platforms on the Databricks platform. You will drive analytics and AI use cases by delivering end-to-end Databricks solutions, Delta Lake data management, and secure data governance.

You will lead data engineering initiatives, collaborate with data scientists and stakeholders, and mentor junior engineers while shaping best practices for

Qualifications

• 7–8 years of hands-on experience building scalable data platforms on Databricks.
• Experience designing, building, and operationalizing data solutions on Databricks for analytics/AI.

Responsibilities

• Architect, develop, productionize and maintain end-to-end Databricks solutions.
• Implement and optimize ETL/ELT processes for structured and semi-structured data.
• Leverage Delta Lake for ACID, versioning and time-travel features.
• Drive Lakehouse adoption to unify data warehousing and AI/ML workloads.
• Configure CI/CD pipelines using Databricks Repos, Asset Bundles and DevOps tools.
• Set up and enforce Unity Catalog for governed data access.
• Design data quality and validation frameworks to ensure trusted data.
• Lead performance tuning for Spark jobs and queries.
• Integrate with external systems (Kafka, Event Hub, REST APIs) for real-time/batch processing.
• Collaborate with data scientists to build feature-rich datasets and reusable assets.
• Troubleshoot and debug complex data workflows in dev/production.
• Mentor junior engineers and promote best practices in data engineering and platform usage.
• Ensure platform security, access controls and governance compliance.

Skills

Apache Spark

Databricks

Delta Lake

PySpark

SQL

Data Engineering

ETL/ELT

Unity Catalog

DevOps

CI/CD

Mosaic AI

Structured Streaming

Tools

Databricks Notebooks

Databricks Jobs

Databricks Repos

Asset Bundles

DevOps tools

Job description

As part of ourInnovation Team, we are seeking aCertified Senior Databricks Engineer / Tech Leadwith 7–8 years of hands-on experience in building scalable data platforms. This role will focus on designing, building, and operationalizing data solutions on theDatabricks platformto accelerate advanced analytics and AI use cases.

Key Responsibilities:

• Architect, develop, productionize and maintain end to end solutions inDatabricks
• Implement and optimizeETL/ELT processesfor structured and semi-structured data
• LeverageDelta Lakefor ACID transactions, data versioning, and time-travel features
• Drive adoption of theLakehouse architectureto unify data warehousing and AI/ML workloads
• ImplementCI/CD pipelinesusingDatabricks Repos,Asset Bundles, and integration withDevOps tools
• Configure and enforceUnity Catalogfor secure, governed access to data assets
• Design and implementdata quality and validation frameworksto ensure trusted data
• Lead performance tuning and optimization efforts forSpark jobs and queries
• Integrate withexternal systemssuch asKafka,Event Hub, andREST APIsfor real-time and batch processing
• Collaborate with data scientists and business stakeholders to build feature-rich datasets and reusable assets
• Troubleshoot and debug complex data workflows in development and production environments
• Guide junior engineers and contribute to best practices in data engineering and platform usage
• Ensure platformsecurity, access controls, and compliance with enterprise data governance standards.

Required Skills:

• Expertise in Apache Spark and Databricks platform
• Experience with Databricks Lakehouse architecture
• Delta Lake concepts
• Proficient in PySpark, SQL, and Delta Lake
• Strong knowledge of Data Engineering concepts
• Experience with data ingestion, ETL/ELT pipelines
• Familiarity with Unity Catalog and data governance
• Hands-on with Databricks Notebooks and Jobs
• CI/CD automation with Databricks Repos and DevOps, Asset Bundles
• Databricks Asset Bundle implementation knowledge
• Strong understanding of performance tuning in Spark
• Data quality and validation framework implementation
• Experience in handling structured, semi-structured data
• Proficient in debugging and troubleshooting
• Collaboration with data scientists and analysts
• Good understanding of security and access control
• Experience withMosaic AIor Databricks ML capabilities
• Exposure tostreaming pipelinesusing Structured Streaming
• Familiarity withdata observabilityandlineage tools.