Senior Databricks Data Engineer

Tekskills Inc.; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tekskills Inc. in Hyderabad, India seeks a Senior Data Engineer with 6-8 years of dedicated experience in data engineering and modern data platform architectures.

You will work with Databricks Lakehouse, Delta Lake, and Unity Catalog, excelling in SQL and PySpark. You will design and maintain scalable batch and real-time data pipelines on Databricks, optimize queries, implement governance, and leverage AWS services.

Qualifications

• 6-8 years of dedicated experience in Data Engineering with a focus on modern data platform architectures.
• Hands-on Databricks Lakehouse, Delta Lake, Unity Catalog implementation.
• Strong SQL and PySpark skills including structured streaming and DataFrame API.
• Experience with AWS cloud services and data warehousing concepts.
• Knowledge of dimensional modeling and star/snowflake schemas.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines on Databricks.
• Optimize complex SQL queries and PySpark jobs for performance and cost efficiency.
• Implement data governance using Databricks Unity Catalog and access controls.
• Design data solutions leveraging AWS services such as S3, IAM, Lambda, EC2.
• Serve as technical liaison to clients, gather requirements, and provide updates.
• Promote best practices in code quality, version control (Git), CI/CD, and testing.

Skills

Databricks Lakehouse

Delta Lake

Unity Catalog

SQL

PySpark

AWS

Data Modeling

Data Warehousing

Star/Snowflake Schemas

Education

Tools

Git

CI/CD

Data Testing Frameworks

Job description

• Experience: 6-8 years of dedicated experience in Data Engineering, with a strong focus on modern data platform architectures.
• Databricks Expertise: Deep hands-on experience with the Databricks Lakehouse platform, Delta Lake, and Unity Catalog implementation.
• Core Languages: Exceptional proficiency in SQL (advanced querying, tuning, data modeling) and PySpark (structured streaming, DataFrame API).
• Cloud Ecosystem: Strong exposure to the AWS cloud platform and its core data/compute infrastructure services.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.

Roles & Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable, reliable, and secure batch and real-time data pipelines using PySpark and SQL within the Databricks platform.
• Architecture & Optimization: Optimize complex SQL queries and PySpark jobs for performance, cost, and memory efficiency.
• Data Governance: Implement and manage data governance, access controls, and data discovery workflows using Databricks Unity Catalog.
• Cloud Integration: Design and deploy data solutions leveraging the AWS Cloud ecosystem (e.g., S3, IAM, Lambda, EC2, CloudWatch, or Redshift).
• Client & Stakeholder Management: Serve as a primary technical point of contact for clients. Gather requirements, present technical solutions, manage expectations, and provide regular project updates.
• Mentorship & Best Practices: Promote best practices in code quality, version control (Git), CI/CD, and data testing frameworks within the engineering team.