Require a Data Engineer in Pune

TestHiring; Pune District; On-site; INR 700,000 - 1,000,000; Full time

Job summary

TestHiring is seeking a Data Engineer to design, develop, and maintain data capabilities within a scalable data lake infrastructure. You will build data pipelines, ensure data quality, and collaborate with cross-functional teams to translate requirements into robust technical solutions.

The role requires hands-on experience with Databricks, Spark, Python, SQL, Hadoop, and Airflow, plus cloud familiarity (AWS/Azure). Strong communication and problem-solving skills are essential.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or related field.
• Proven experience as a Data Engineer / Scientist or similar role.
• Deep understanding of data engineering, ETL/ELT, data warehousing and data modeling.
• Strong data integration techniques and data quality management.
• Hands-on experience with Databricks, Spark, Python, SQL, Hadoop, Airflow.
• Familiarity with AWS or Azure.
• Excellent analytical and problem-solving skills with innovative data solutions.
• Strong interpersonal and communication skills, and ability to work with dispersed teams.

Responsibilities

• Design, develop, and maintain data capabilities and infrastructure for Sustainable Tech Internal Data Lake.
• Create data pipelines, transfers, and compliant infrastructure for on-premise/cloud environments.
• Identify data gaps across initiatives and provide SME support for remediation.
• Collaborate with technical teams and business stakeholders to translate data requirements into solutions.
• Work with large datasets ensuring data quality, accuracy, and performance.
• Implement data transformation, integration, and validation for analytics/BI and reporting.
• Optimize data pipelines for speed, reliability, and efficiency.
• Implement best practices for data storage, retrieval, and archival for accessibility and security.
• Troubleshoot data-related issues and identify root causes.
• Document data processes, lineage, and technical specifications.
• Participate in code reviews and adhere to coding standards.
• Collaborate with DevOps to automate deployment and monitoring of data pipelines.
• Additional tasks as required.

Skills

Analytical skills

Interpersonal skills

Communication skills

Education

Bachelor's degree in Computer Science/Engineering/Data Science

Tools

Databricks

Spark

Python

SQL

Hadoop

Airflow

AWS

Azure

Job description

Key Responsibilities:

• Design, develop, and maintain new data capabilities and infrastructure for Mastercard's Sustainable Technology Internal Data Lake.
• Create new data pipelines, data transfers, and compliance-oriented infrastructure to facilitate seamless data utilization within on-premise/cloud environments.
• Identify existing data capability and infrastructure gaps or opportunities within and across initiatives and provide subject matter expertise in support of remediation.
• Collaborate with technical teams and business stakeholders to understand data requirements and translate them into technical solutions.
• Work with large datasets, ensuring data quality, accuracy, and performance.
• Implement data transformation, integration, and validation processes to support analytics/BI and reporting needs.
• Optimize and fine-tune data pipelines for improved speed, reliability, and efficiency.
• Implement best practices for data storage, retrieval, and archival to ensure data accessibility and security.
• Troubleshoot and resolve data-related issues, collaborating with the team to identify root causes.
• Document data processes, data lineage, and technical specifications for future reference.
• Participate in code reviews, ensuring adherence to coding standards and best practices.
• Collaborate with DevOps teams to automate deployment and monitoring of data pipelines.
• Additional tasks as required.

Requirements

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
• Proven experience as a Data Engineer / Scientist or similar role.
• Deep understanding & expertise in data engineering, ETL/ELT processes, data warehousing, and data modeling.
• Strong command of data integration techniques and data quality management.
• Hands-on experience with data technologies such as Databricks, Spark, Python, SQL, Hadoop, Airflow.
• Familiarity with cloud platforms and services, such as AWS or Azure.
• Excellent analytical, problem-solving skills and ability to provide innovative data solutions.
• Exceptional interpersonal skills with proven experience in relationship building and partnering, must work well in both team/individual settings and must be able to work with a geographically dispersed team.
• Strong written and oral communication skills. Attention to detail is a must.
• Motivated self-starter with ability to excel at multi-tasking in a fast-paced environment and able to function under pressure with a high degree of initiative to drive results.
• Ability to quickly learn and implement new technologies and perform POC to explore best solutions for problem statements.
• Flexibility to work as a member of a matrix-based diverse and geographically distributed project team.

• 0.6 -1.5 years of experience in Data Engineering or Warehouse-related projects.
• Expertise in Data Engineering and Data Analysis: implementing multiple end-to-end Data Engineering or Warehouse projects in Big Data environment.
• Experience building data pipelines through Spark with Scala/Python/Java in Databricks or Hadoop environment
• Experience working with databases like MS SQL Server or Oracle, and strong SQL knowledge.
• Experience automating data flow processes in a Big Data environment using Airflow or similar tools.
• Experience working in Agile teams.