Lead Data Engineer

KPI Partners; Bengaluru; In loco; INR 3.000.000 - 5.000.000; Tempo pieno

Descrizione del lavoro

KPI Partners is seeking a seasoned Databricks Data Engineer to design, develop, and optimize scalable data pipelines using Databricks, PySpark, Python and SQL. The role emphasizes Spark performance tuning, Lakehouse architecture with Delta Lake, and cloud platforms across two providers.

You will implement CI/CD and MLOps, and lead customer engagements for international clients. The ideal candidate collaborates with offshore/onshore teams, providing technical guidance and delivering

Competenze

• 8–12 years of experience in data engineering or analytics
• Strong hands-on expertise in Databricks, PySpark, Python, SQL
• Experience with Spark performance tuning and cost optimization
• Experience with Lakehouse architecture and Delta Lake
• Cloud platform experience across at least two of AWS, Azure, GCP
• Familiarity with CI/CD and MLOps workflows
• Consulting or customer-facing experience is a plus

Mansioni

• Design and develop scalable data pipelines using Databricks and PySpark.
• Work on Apache Spark and distributed data processing.
• Optimize Spark jobs for performance and cost efficiency.
• Implement CI/CD pipelines and MLOps workflows.
• Work on Lakehouse architecture and production deployments.
• Collaborate with international clients and offshore/onshore teams.
• Provide technical guidance and lead customer engagements.

Conoscenze

Python

SQL

Spark

Delta Lake

Lakehouse Architecture

Cloud Platforms

CI/CD

MLOps

Consulting

Strumenti

Databricks

PySpark

Apache Spark

Descrizione del lavoro

KPI Partners is a global data, analytics, and AI consulting company that helps businesses modernize their data platforms and build advanced analytics solutions. The company works with technologies such as Databricks, Azure, AWS, GCP, and Snowflake.

Website: https://www.kpipartners.com/

About the Role – Databricks Data Engineer

Looking for an experienced Databricks Data Engineer with 8–12 years of experience to design, develop, and optimize scalable data pipelines. The candidate should have strong hands-on expertise in Databricks, PySpark, Apache Spark, Python, SQL, and cloud platforms.

• Design and develop scalable data pipelines using Databricks and PySpark.
• Work on Apache Spark and distributed data processing.
• Optimize Spark jobs for performance and cost efficiency.
• Implement CI/CD pipelines and MLOps workflows.
• Work on Lakehouse architecture and production deployments.
• Collaborate with international clients and offshore/onshore teams.
• Provide technical guidance and lead customer engagements.

Must-Have Skills

• Python and SQL
• Spark Performance Tuning
• Delta Lake and Lakehouse Architecture
• AWS, Azure, GCP (at least 2 cloud platforms)
• CI/CD and MLOps
• Consulting or customer-facing experience

Experience: 8–12 years