Senior Databricks Data Engineer

Tekskills Inc.; Chennai; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Tekskills Inc. seeks a Senior Data Engineer in Chennai to design, build and optimize scalable data pipelines on the Databricks Lakehouse platform.

You will work with Delta Lake and Unity Catalog, streaming and batch processing with PySpark and SQL, and collaborate with clients to deliver robust data solutions on AWS infrastructure. The role requires 6–8 years of dedicated data engineering experience, strong data modeling skills, and a drive to implement governance and best practices across the

Qualifications

• 6–8 years of dedicated experience in Data Engineering.
• Deep hands-on experience with Databricks Lakehouse, Delta Lake and Unity Catalog.
• Proficiency in SQL (advanced querying, modeling) and PySpark (structured streaming, DataFrame API).
• Strong exposure to AWS cloud services and data/compute infra.
• Solid understanding of data warehousing concepts and dimensional modeling.

Responsibilities

• Design, develop and maintain scalable batch and real-time data pipelines in Databricks.
• Optimize SQL queries and PySpark jobs for performance and cost efficiency.
• Implement data governance and access controls using Unity Catalog.
• Design data solutions leveraging AWS Cloud services such as S3, IAM, Lambda, Redshift.
• Engage with clients as technical lead, gather requirements and provide updates.
• Promote best practices in code quality, versioning, CI/CD, and data testing.

Skills

SQL

PySpark

Databricks

Tools

Databricks Lakehouse

Delta Lake

Unity Catalog

AWS (S3, IAM, Lambda)

Job description

• Experience: 6-8 years of dedicated experience in Data Engineering, with a strong focus on modern data platform architectures.
• Databricks Expertise: Deep hands-on experience with the Databricks Lakehouse platform, Delta Lake, and Unity Catalog implementation.
• Core Languages: Exceptional proficiency in SQL (advanced querying, tuning, data modeling) and PySpark (structured streaming, DataFrame API).
• Cloud Ecosystem: Strong exposure to the AWS cloud platform and its core data/compute infrastructure services.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.

Roles & Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable, reliable, and secure batch and real-time data pipelines using PySpark and SQL within the Databricks platform.
• Architecture & Optimization: Optimize complex SQL queries and PySpark jobs for performance, cost, and memory efficiency.
• Data Governance: Implement and manage data governance, access controls, and data discovery workflows using Databricks Unity Catalog.
• Cloud Integration: Design and deploy data solutions leveraging the AWS Cloud ecosystem (e.g., S3, IAM, Lambda, EC2, CloudWatch, or Redshift).
• Client & Stakeholder Management: Serve as a primary technical point of contact for clients. Gather requirements, present technical solutions, manage expectations, and provide regular project updates.
• Mentorship & Best Practices: Promote best practices in code quality, version control (Git), CI/CD, and data testing frameworks within the engineering team.