Senior Databricks Data Engineer

Tekskills Inc.; Mumbai; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Tekskills Inc. in Mumbai seeks a senior Data Engineer to build and optimize modern data platforms using Databricks Lakehouse, Delta Lake, and Unity Catalog. You will develop robust batch and real-time pipelines in SQL and PySpark on AWS.

The role emphasizes data governance, cost-aware performance tuning, and collaboration with clients to deliver scalable data solutions. 6–8 years of dedicated experience and strong data warehousing knowledge are required.

Qualifications

• 6-8 years of dedicated experience in Data Engineering with focus on modern data platforms.
• Deep hands-on Databricks Lakehouse experience, Delta Lake and Unity Catalog implementation.
• Strong SQL and PySpark skills for advanced querying, tuning, and data modeling.
• Exposure to AWS cloud services and core data/compute infrastructure.

Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable batch and real-time pipelines using PySpark and SQL within Databricks.
• Architecture & Optimization: Improve complex SQL queries and PySpark jobs for performance and cost efficiency.
• Data Governance: Implement access controls and data discovery workflows using Unity Catalog.
• Cloud Integration: Design data solutions leveraging AWS services like S3, IAM, Lambda, EC2, CloudWatch, and Redshift.
• Client & Stakeholder Management: Act as technical lead, gather requirements, present solutions, and provide project updates.
• Mentorship & Best Practices: Promote code quality, Git usage, CI/CD, and data testing within the team.

Skills

SQL

PySpark

Databricks

Delta Lake

Unity Catalog

AWS

Data Modeling

Data Warehousing

Tools

Git

CI/CD

Job description

• Experience: 6-8 years of dedicated experience in Data Engineering, with a strong focus on modern data platform architectures.
• Databricks Expertise: Deep hands-on experience with the Databricks Lakehouse platform, Delta Lake, and Unity Catalog implementation.
• Core Languages: Exceptional proficiency in SQL (advanced querying, tuning, data modeling) and PySpark (structured streaming, DataFrame API).
• Cloud Ecosystem: Strong exposure to the AWS cloud platform and its core data/compute infrastructure services.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.

Roles & Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable, reliable, and secure batch and real-time data pipelines using PySpark and SQL within the Databricks platform.
• Architecture & Optimization: Optimize complex SQL queries and PySpark jobs for performance, cost, and memory efficiency.
• Data Governance: Implement and manage data governance, access controls, and data discovery workflows using Databricks Unity Catalog.
• Cloud Integration: Design and deploy data solutions leveraging the AWS Cloud ecosystem (e.g., S3, IAM, Lambda, EC2, CloudWatch, or Redshift).
• Client & Stakeholder Management: Serve as a primary technical point of contact for clients. Gather requirements, present technical solutions, manage expectations, and provide regular project updates.
• Mentorship & Best Practices: Promote best practices in code quality, version control (Git), CI/CD, and data testing frameworks within the engineering team.