Data Engineer

Minivel; Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Minivel in Bengaluru seeks a Databricks Developer/Data Engineer to design and optimize scalable data processing solutions using Python, PySpark, and Spark. You will collaborate with data scientists, engineers, and stakeholders to deliver high-quality data products.

The role requires strong SQL, experience in Databricks, and a solid foundation in data engineering concepts, data validation, and pipeline development.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, IT, or related field.
• Proven experience as a Databricks Developer, Data Engineer, or similar role.
• Strong hands-on expertise in Apache Spark and PySpark.
• Strong proficiency in SQL and hands-on experience with relational databases.

Responsibilities

• Design, develop, and maintain scalable data processing applications using Python, PySpark, and Spark.
• Develop and optimize data pipelines and workflows on Databricks.
• Collaborate with data engineers, data scientists, business stakeholders, and other technical teams to deliver high-quality solutions.
• Ensure data integrity, quality, performance, and reliability across data processing pipelines.
• Write clean, maintainable, scalable, and efficient code following established coding standards and best practices.
• Perform data analysis and implement data validation and quality checks.
• Monitor, troubleshoot, and optimize performance issues across data workflows and pipelines.
• Work with relational databases and develop efficient SQL queries for data extraction and transformation.
• Participate in code reviews, testing, deployment, and continuous improvement of data engineering solutions.
• Use Git/version control and follow established software development and deployment practices.

Skills

Python

SQL

Apache Spark

PySpark

Data analysis

Education

Bachelor's or Master's in Computer Science, Engineering, Information Technology

Tools

Databricks

Git

Scala

Job description

Key Responsibilities

• Design, develop, and maintain scalable data processing applications using Python, PySpark, and Spark.
• Develop and optimize data pipelines and workflows on Databricks.
• Collaborate with data engineers, data scientists, business stakeholders, and other technical teams to understand requirements and deliver high-quality solutions.
• Ensure data integrity, quality, performance, and reliability across data processing pipelines.
• Write clean, maintainable, scalable, and efficient code following established coding standards and best practices.
• Perform data analysis and implement appropriate data validation and quality checks.
• Monitor, troubleshoot, and optimize performance issues across data workflows and pipelines.
• Work with relational databases and develop efficient SQL queries for data extraction and transformation.
• Participate in code reviews, testing, deployment, and continuous improvement of data engineering solutions.
• Use Git/version control and follow established software development and deployment practices.

Required Skills & Qualifications

• Bachelors or Master’s degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Databricks Developer, Data Engineer, or similar role.
• Strong hands‑on expertise in Apache Spark and PySpark.
• Strong programming skills in Python; experience with Scala is an advantage.
• Strong proficiency in SQL and hands‑on experience with relational databases.
• Practical experience developing and optimizing data pipelines and data processing applications.
• Familiarity with Git and version control systems.
• Strong understanding of data engineering concepts, data transformation, and data validation.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.