Senior Databricks Data Engineer

Tekskills Inc.; Kolkata District; On-site; INR 3,000,000 - 3,800,000; Full time

Job summary

Tekskills Inc. in Kolkata seeks a senior Data Engineer to shape data platform architectures using Databricks Lakehouse, Delta Lake, and Unity Catalog. You will leverage SQL and PySpark to build scalable pipelines and optimize performance within the AWS ecosystem.

You will drive data governance, collaborate with clients and stakeholders, and mentor engineers while enforcing CI/CD and testing best practices in a fast-paced environment.

Qualifications

• 6–8 years of dedicated data engineering experience.
• Deep hands-on Databricks Lakehouse, Delta Lake, and Unity Catalog.
• Strong SQL and PySpark proficiency for data wrangling and streaming.
• Exposure to AWS cloud services and data warehousing concepts.

Responsibilities

• Data Pipeline Development: design and maintain scalable batch and real-time pipelines in Databricks.
• Architecture & Optimization: tune complex SQL queries and PySpark jobs for performance and cost.
• Data Governance: implement access controls and data discovery with Unity Catalog.
• Cloud Integration: design data solutions leveraging AWS services (S3, IAM, Lambda, Redshift).
• Client & Stakeholder Management: gather requirements and provide regular project updates.
• Mentorship & Best Practices: promote code quality, version control, CI/CD, and testing.

Skills

Databricks Lakehouse

SQL

PySpark

Unity Catalog

Delta Lake

AWS

Data Modeling

Data Governance

Git

CI/CD

Tools

Git

CI/CD

Job description

• Experience: 6-8 years of dedicated experience in Data Engineering, with a strong focus on modern data platform architectures.
• Databricks Expertise: Deep hands-on experience with the Databricks Lakehouse platform, Delta Lake, and Unity Catalog implementation.
• Core Languages: Exceptional proficiency in SQL (advanced querying, tuning, data modeling) and PySpark (structured streaming, DataFrame API).
• Cloud Ecosystem: Strong exposure to the AWS cloud platform and its core data/compute infrastructure services.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.

Roles & Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable, reliable, and secure batch and real-time data pipelines using PySpark and SQL within the Databricks platform.
• Architecture & Optimization: Optimize complex SQL queries and PySpark jobs for performance, cost, and memory efficiency.
• Data Governance: Implement and manage data governance, access controls, and data discovery workflows using Databricks Unity Catalog.
• Cloud Integration: Design and deploy data solutions leveraging the AWS Cloud ecosystem (e.g., S3, IAM, Lambda, EC2, CloudWatch, or Redshift).
• Client & Stakeholder Management: Serve as a primary technical point of contact for clients. Gather requirements, present technical solutions, manage expectations, and provide regular project updates.
• Mentorship & Best Practices: Promote best practices in code quality, version control (Git), CI/CD, and data testing frameworks within the engineering team.