Databricks Engineer

KPI Partners; Hyderabad, Pune District, Bengaluru; Presencial; INR 1.800.000 - 2.400.000; Jornada completa

Descripción de la vacante

KPI Partners is seeking a Lead Databricks Engineer to drive Databricks-based data solutions, pipelines, and MLOps practices. The role emphasizes collaboration with data scientists, CI/CD implementation, and performance optimization within the Databricks ecosystem.

Ideal candidates will have strong Databricks and PySpark experience, plus a background in CI/CD and mentoring junior engineers. This is a senior, hands-on technical leadership position at KPI Partners.

Formación

• Bachelor or higher in Computer Science, Engineering or related field.
• 5+ years in data engineering or related role, with at least 2 years in Databricks.
• Proven experience using Pyspark for data processing.
• Experience implementing CI/CD in data projects.
• Knowledge of machine learning concepts and MLOps is a plus.

Responsabilidades

• Lead design and implementation of Databricks-based data solutions.
• Develop, test, and deploy data pipelines using Pyspark.
• Collaborate with data scientists to implement MLOps for deploying models.
• Establish and maintain CI/CD pipelines for high-quality code delivery.
• Optimize data workflows and performance within Databricks environment.
• Mentor junior engineers on best practices and emerging technologies.
• Translate business requirements into technical specifications.

Conocimientos

Databricks

Pyspark

CI/CD

MLOps

Problem solving

Teamwork

Educación

Bachelors (CS/Engineering)

Herramientas

CI/CD tools

Descripción del empleo

Description

Job Title: Lead Databricks Engineer (Databricks)

KPI Partners is seeking a skilled and experienced Lead Databricks Engineer to join our dynamic team. The ideal candidate will have a strong background in Databricks, Pyspark, CI/CD practices, and MLOps. This role will be pivotal in developing data solutions that drive business insights and enhance operational efficiency.

Key Responsibilities:

• Lead the design and implementation of Databricks-based data solutions.
• Develop, test, and deploy data processing pipelines using Pyspark.
• Collaborate with data scientists and analysts to implement MLOps practices for deploying machine learning models.
• Establish and maintain CI/CD pipelines to automate the deployment process and ensure high-quality code delivery.
• Optimize data workflows and improve system performance within the Databricks environment.
• Mentor and guide junior engineers and team members in best practices and emerging technologies.
• Work closely with stakeholders to understand business requirements and translate them into technical specifications.

Required Skills:

• Proficient in Databricks and its ecosystem.
• Strong experience with Pyspark for data processing and analysis.
• Solid understanding of CI/CD practices and tools.
• Familiarity with MLOps concepts and their application in real-world scenarios.
• Excellent problem-solving skills and attention to detail.
• Ability to work collaboratively in a fast-paced, team-oriented environment.

Qualifications:

• Bachelors degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering or a related role, with at least 2 years of experience specifically in Databricks.
• Proven experience in leveraging Pyspark for data processing.
• Experience implementing CI/CD practices in data projects.
• Knowledge of machine learning concepts and MLOps practices is a plus.

If you are passionate about data engineering and looking to take your career to the next level with KPI Partners, Join us in driving innovation through data!