Lead Data Engineer

Saur Energy International; Bengaluru; Vor Ort; INR 4.000.000 - 7.000.000; Vollzeit

Zusammenfassung

Saur Energy International in Bengaluru seeks a senior Data Engineer to design and operate scalable Databricks data pipelines and Lakehouse solutions, focusing on performance, governance, and cost optimization.

You will mentor engineers, drive CI/CD for Databricks assets, implement Unity Catalog and Delta Lake design, and collaborate with cross-functional teams to deliver reliable data services in production.

Qualifikationen

• Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
• 6+ years of data engineering experience with 2+ years hands-on Databricks in enterprise settings.
• Deep understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and Workflow orchestration.
• Proven ability to tune Spark workloads for cost and performance at production scale.
• Advanced Python (PySpark) and SQL skills.
• Working knowledge of CI/CD practices and DevOps principles applied to data workloads.
• Experience with observability tooling for Databricks.

Aufgaben

• Design and develop scalable data pipelines and Lakehouse solutions on Databricks.
• Tune Databricks workloads for performance and cost, including cluster sizing, query optimization, and Delta Lake table design.
• Establish and enforce best practices for partitioning, clustering, and workload isolation.
• Track performance trends, identify high-cost queries, and partner with source teams and end users to resolve long-running loads.
• Design and operationalize Unity Catalog for data governance — access control, lineage, and security.
• Build monitoring and self-healing automation using Databricks-native AI and agentic capabilities.
• Drive CI/CD workflows for Databricks assets, setting DevOps best practices for deployment and release management.
• Lead design reviews and mentor Data Engineers on Databricks best practices and AI-native features.
• Own Databricks vendor coordination — case management, escalations, and release adoption strategy.

Kenntnisse

Python (PySpark)

SQL

CI/CD

DevOps

Observability tooling

Performance tuning

Platform ownership

Communication

Ausbildung

Bachelor's or Master's degree in Computer Science/IT or equivalent

Tools

Databricks

Delta Lake

Unity Catalog

DBSQL

Databricks Serverless Compute

Jobbeschreibung

Key Responsibilities:

• Design and develop scalable data pipelines and Lakehouse solutions on Databricks.
• Tune Databricks workloads for performance and cost, including cluster sizing, query optimization, and Delta Lake table design.
• Establish and enforce best practices for partitioning, clustering, and workload isolation.
• Track performance trends, identify high-cost queries, and partner with source teams and end users to resolve long-running loads.
• Design and operationalize Unity Catalog for data governance — access control, lineage, and security.
• Build monitoring and self-healing automation using Databricks-native AI and agentic capabilities.
• Drive CI/CD workflows for Databricks assets, setting DevOps best practices for deployment and release management.
• Lead design reviews and mentor Data Engineers on Databricks best practices and AI-native features.
• Own Databricks vendor coordination — case management, escalations, and release adoption strategy.

What Success Looks Like (First 6–12 Months)

• Within 6–12 months, you'll define the platform's tuning and governance standards, lead design reviews, mentor junior engineers, and shape the AI-native operations roadmap.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
• 6+ years of data engineering experience with 2+ years hands-on Databricks in enterprise settings.
• Deep understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and Workflow orchestration.
• Proven ability to tune Spark workloads for cost and performance at production scale.
• Advanced Python (PySpark) and SQL skills.
• Working knowledge of CI/CD practices and DevOps principles applied to data workloads.
• Experience with observability tooling for Databricks.

Preferred Qualifications:

• Experience with Databricks-native AI capabilities and agentic frameworks.
• Familiarity with Databricks Serverless Compute and DBSQL performance tuning.
• A Databricks Certified Professional.
• Exposure toInfrastructure-as-Codeis a plus.

Competencies:

• Performance-engineeringmindset — measures, tunes, and re-measures.
• Curiosity for AI-native operations and continuous automation.
• Strong sense of platform ownership — quality, cost, and reliability.
• Effective communication with engineering peers, vendors, and business stakeholders.
• Influence outcomes across source teams, vendors, and business stakeholders without direct authority.