Platform Engineer - Azure

KPI Partners; Hyderabad, Pune District, Bengaluru; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

KPI Partners in Hyderabad is seeking an experienced Data Engineer to design scalable Spark-based data pipelines, optimize jobs, and automate workflows. You will work on Delta Lake management, modern lakehouse architectures, and data quality across large datasets.

The role emphasizes CI/CD for data artifacts, collaboration with cross-functional teams, and hands-on scripting in Python. 6+ years of experience, strong problem-solving, and excellent communication are essential.

Qualifications

• 6+ years of hands-on experience in data engineering roles.
• Strong hands-on experience with Apache Spark (PySpark or SparkSQL) and Spark job optimization techniques.
• Proficiency in Python for scripting, automation, and data manipulation.
• Experience working with APIs and other source systems.
• Basic understanding of DevOps principles and experience with CI/CD tools.
• Familiarity with cloud platforms such as Azure.
• Experience with version control systems like Git.
• Ability to work independently and collaboratively in a fast-paced environment.
• Excellent problem-solving and communication skills.

Responsibilities

• Design, develop, and optimize large-scale data pipelines using Apache Spark.
• Knowledge on Modern Lakehouse Architecture and management of Delta tables.
• Optimize Spark jobs for performance, scalability, and cost-efficiency across large datasets.
• Implement performance tuning strategies for Spark jobs to ensure efficient data processing.
• Develop automation scripts in Python to streamline integration, deployment, and maintenance of data engineering workflows.
• Collaborate with cross-functional teams to automate manual tasks and improve operational efficiency.
• Contribute to CI/CD pipelines for data engineering artifacts using tools like Azure DevOps, GitHub Actions.
• Ensure data quality, reliability, and security across all stages of the data lifecycle.
• Monitor and troubleshoot production data pipelines and resolve issues proactively.
• Document processes, workflows, and best practices for data engineering operations.

Skills

Apache Spark

PySpark

SparkSQL

Python

APIs

CI/CD tools

Azure

Git

Communication

Problem solving

Tools

Delta Lake

Azure DevOps

GitHub Actions

Git

Job description

Description

We at KPI Partners are seeking a skilled and proactive Data Engineer with strong expertise in Apache Spark, Python, and DevOps practices to join our data engineering team. The ideal candidate will be responsible for building scalable data pipelines, optimizing Spark jobs, automating data workflows, and contributing to CI/CD processes for data engineering artifacts.

Key Responsibilities

• Design, develop, and optimize large-scale data pipelines using Apache Spark.
• Knowledge on Modern Lakehouse Architecture and management of Delta tables.
• Optimize Spark jobs for performance, scalability, and cost-efficiency across large datasets.
• Implement performance tuning strategies for Spark jobs to ensure efficient data processing.
• Develop automation scripts in Python to streamline integration, deployment, and maintenance of data engineering workflows.
• Collaborate with cross-functional teams to automate manual tasks and improve operational efficiency.
• Contribute to CI/CD pipelines for data engineering artifacts using tools like Azure DevOps, GitHub Actions.
• Ensure data quality, reliability, and security across all stages of the data lifecycle.
• Monitor and troubleshoot production data pipelines and resolve issues proactively.
• Document processes, workflows, and best practices for data engineering operations.

Required Skills & Qualifications

• 6+ years of hands-on experience in data engineering roles.
• Strong hands-on experience with Apache Spark (PySpark or SparkSQL) and Spark job optimization techniques.
• Proficiency in Python for scripting, automation, and data manipulation.
• Should have experience in working with APIs and other Source systems.
• Basic understanding of DevOps principles and experience with CI/CD tools.
• Familiarity with cloud platforms such as Azure.
• Experience with version control systems like Git.
• Ability to work independently and collaboratively in a fast-paced environment.
• Excellent problem-solving and communication skills.