Senior Databricks Data Engineer

Tekskills Inc.; Delhi; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Tekskills Inc. in Delhi seeks a Senior Data Engineer with 6–8 years of dedicated data engineering experience. The role emphasizes Databricks Lakehouse, Delta Lake, Unity Catalog, SQL tuning, and PySpark within an AWS-driven data stack.

You will design and maintain scalable batch and real-time pipelines on Databricks, optimize queries for performance and cost, govern data with Unity Catalog, and collaborate with clients to deliver robust data solutions.

Qualifications

• 6–8 years of dedicated experience in Data Engineering.
• Hands-on with Databricks Lakehouse, Delta Lake, and Unity Catalog.
• Proficient in SQL (advanced querying, tuning) and PySpark.
• Strong exposure to AWS cloud services.
• Solid data warehousing and dimensional modeling concepts.

Responsibilities

• Data Pipeline Development: design, build, and maintain scalable batch and real-time pipelines on Databricks.
• Architecture & Optimization: tune SQL queries and PySpark jobs for performance and cost efficiency.
• Data Governance: implement Unity Catalog access controls and data discovery workflows.
• Cloud Integration: leverage AWS services (S3, IAM, Lambda, EC2, CloudWatch, Redshift).
• Client & Stakeholder Management: gather requirements, present solutions, and provide project updates.
• Mentorship & Best Practices: promote code quality, Git workflows, CI/CD, and data testing.

Skills

Databricks

SQL

PySpark

AWS

Data pipelines

Tools

Unity Catalog

Delta Lake

Job description

• Experience: 6-8 years of dedicated experience in Data Engineering, with a strong focus on modern data platform architectures.
• Databricks Expertise: Deep hands-on experience with the Databricks Lakehouse platform, Delta Lake, and Unity Catalog implementation.
• Core Languages: Exceptional proficiency in SQL (advanced querying, tuning, data modeling) and PySpark (structured streaming, DataFrame API).
• Cloud Ecosystem: Strong exposure to the AWS cloud platform and its core data/compute infrastructure services.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.

Roles & Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable, reliable, and secure batch and real-time data pipelines using PySpark and SQL within the Databricks platform.
• Architecture & Optimization: Optimize complex SQL queries and PySpark jobs for performance, cost, and memory efficiency.
• Data Governance: Implement and manage data governance, access controls, and data discovery workflows using Databricks Unity Catalog.
• Cloud Integration: Design and deploy data solutions leveraging the AWS Cloud ecosystem (e.g., S3, IAM, Lambda, EC2, CloudWatch, or Redshift).
• Client & Stakeholder Management: Serve as a primary technical point of contact for clients. Gather requirements, present technical solutions, manage expectations, and provide regular project updates.
• Mentorship & Best Practices: Promote best practices in code quality, version control (Git), CI/CD, and data testing frameworks within the engineering team.