Principal Data Engineer

Quest Global; Maharashtra; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Quest Global is seeking a Data Architect to design and implement data-based solutions within the Mastercard technology portfolio. You will play an essential role in translating business needs into technical solutions, working closely with various stakeholders while ensuring the alignment of business goals with technical execution.

Ideal candidates will have experience with modern data platforms, particularly Cloudera, and will excel in a hands-on technical role while guiding teams to build robust data architectures and pipelines.

Qualifications

• Proven experience as data architect and engineer on both on-prem and cloud-based platforms.
• Hands-on experience with modern Data Platforms, specifically Cloudera Data Platform.
• Strong knowledge of performance engineering and scalability.

Responsibilities

• Architect Data based solutions aligned with business needs.
• Collaborate with technical stakeholders to translate requirements into architecture.
• Drive adoption of software engineering best practices.

Skills

Data architecture

Cloud-based platforms

PySpark

ETL processes

Data governance

Tools

Cloudera Data Platform

Apache Airflow

SQL

Job description

Responsibilities & Qualifications

• Architect Data based solutions (foundational and analytical data product) based on business need and in line with the Mastercard technology portfolio, patterns and standards.
• Collaborate closely with finance, business and technical stakeholders to translate requirements into technical data architecture and solution.
• Hands‑on, with the ability to write/develop complex code and also help the engineering teams.
• Exposure to building Data contracts and Data products.
• Partner with data and engineering teams to build robust data pipelines, data modeling, and ensure seamless integration across diverse systems and domains.
• Facilitate trade‑off discussions with both business and technical stakeholders to balance priorities and make informed decisions.
• Ability to choose between different technical options considering all aspects of engineering principles.
• Ensure alignment between business goals and technical execution, making sure features and solutions meet business requirements and customer needs.
• Participate in sprint planning, retrospectives, and other agile ceremonies to ensure the team is aligned and delivering efficiently.
• Drive the adoption of best practices in software engineering, including code reviews, testing, and continuous integration/continuous delivery (CI/CD).
• Optimize the cost/benefit of software features and architecture, ensuring scalability, performance, and operational efficiency.
• Act as leader of Data platform / technology, providing guidance on complex technical challenges and driving resolution, ensuring solutions align with Mastercard’s engineering and data principles, and technical policies.
• Identify opportunities for process improvements, helping to streamline workflows and enhance team productivity.
• Mentor and guide engineers across various experience levels, helping them grow technically and improve their software and data engineering skills.

Work Experience

• Proven experience as data architect and engineer, both on on‑prem and cloud‑based data platforms (traditional to modern platforms) with a strong focus on delivering large‑scale projects in an agile environment.
• Work with the engineering team to help design, develop and implement large scale, high‑volume, high‑performance, highly available, scalable data platform and pipelines for the Lakehouse being built on on‑prem Cloudera Data Platform - CDP. Ability to handle large‑scale data processing and distributed computing across massive datasets.
• Must - Hands‑on experience with deep understanding and experience with modern Data Platforms – Cloudera Data Platform (CDP).
• Strong hands‑on experience with PySpark.
• Experience with Cloudera Data Platform (CDE, CDW, Ozone, Airflow, SDX), Apache Ranger. Deep understanding of distributed data systems and Hive Metastore.
• Experience and understanding of cataloging, lineage, and governance.
• Experience / understanding Open Data Contract Standard (ODCS) and its implementation.
• Experience working with SQL, Iceberg open table and Parquet file format, and partitioning/bucketing strategies.
• Experience with data lifecycle management, including ingestion, ETL, pruning, modelling, and governance, within highly regulated environments.
• Experience with performance engineering, ensuring systems are built to scale and meet varying demands.
• Knowledge of security best practices and experience in ensuring the secure development of applications.
• Nice to have:
• Experience modernising enterprise finance systems or regulated environments.
• Knowledge of CI/CD, data engineering best practices.
• Understanding of financial data structures, accounting processes, or reconciliation workflows.
• Prior experience with financial systems, such as Oracle Financials, Oracle Fusion Cloud, and Hyperion, with experience optimising their integration into broader data ecosystems will be a plus.