Advisor - Cloud Data architect

enGen Global; Chennai District; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

enGen Global in Chennai, India seeks an experienced Lead Data Architect to design and implement a next-generation enterprise data platform on Google Cloud Platform (GCP). The role focuses on architecture roadmaps, technical design leadership, and delivering scalable, secure data solutions for strategic business objectives.

The candidate should have deep knowledge of data warehousing, data lake concepts, and hands-on experience with PySpark, Dataproc, BigQuery, and related open-source

Qualifications

• 12 to 18 years of progressive experience in data architecture or data engineering.
• 3+ years hands-on experience designing large-scale data platforms on GCP.
• Deep expertise with BigQuery, Dataproc, Dataflow, and other GCP data services.
• Strong PySpark experience for batch and streaming processing.
• Experience with Kafka, Pub/Sub, and stream processing patterns.
• Knowledge of data warehousing, data lakes, and data lakehouse concepts.
• Hands-on with Apache Iceberg and versioned data schemas.
• Experience with data governance, data quality, and metadata management.
• Security best practices in cloud data environments.
• Familiarity with Terraform and CI/CD pipelines for data solutions.

Responsibilities

• Design end-to-end data architectures covering ingestion, storage, processing, and consumption layers.
• Lead selection and integration of GCP services (BigQuery, Dataproc, Dataflow) and open-source tech.
• Develop standards and patterns for data platform development and operations.
• Drive robust, scalable data pipelines using PySpark on Dataproc for large-scale processing.
• Use dbt for structured, version-controlled data modeling within BigQuery.
• Implement Iceberg-based storage with schema evolution and time travel capabilities.
• Establish data quality frameworks and governance aligned with regulatory requirements (HIPAA, GDPR).
• Collaborate with governance teams to enforce policies and metadata management.
• Set up monitoring and alerting for data pipelines and platform health.
• Provide technical leadership and mentorship to data engineers and teams.
• Communicate complex concepts clearly to technical and non-technical stakeholders.

Skills

GCP BigQuery

PySpark

Dataproc

Dataflow

dbt

Iceberg

Starburst/Trino

BigLake Metastore

Data Warehousing

Streaming

Kafka

Pub/Sub

Terraform

CI/CD

Security best practices

Education

Bachelor's degree or higher

Tools

Databricks

Iceberg

BigQuery

Job description

Job Summary

We are seeking an experienced and visionary Lead Data Architect to design, build, and optimize our next-generation enterprise data platform on Google Cloud Platform (GCP). This pivotal role will be responsible for defining the architectural roadmap, leading the technical design, and ensuring the successful implementation of robust, scalable, and secure data solutions that support our strategic business objectives. The ideal candidate will have deep expertise in modern data warehousing, data lake, and data lake house concepts, with hands-on experience in advanced GCP data services, PySpark / Databricks and related open-source technologies

Responsibilities

• Design end-to-end data architectures, including data ingestion, transformation, storage, processing, and consumption layers.
• Lead the selection and integration of appropriate GCP services (e.g., BigQuery, Dataproc, Cloud Storage, Dataflow) and open-source technologies.
• Develop architectural standards, patterns, and best practices for data platform development and operations.
• Design and drive the implementation of robust, scalable, and efficient data pipelines using PySpark on Dataproc for large-scale data processing.
• Utilize data transformation tools like dbt (data build tool) for efficient and version-controlled data modeling and transformation within BigQuery.
• Implement and manage data storage solutions leveraging Apache Iceberg for open table formats, enabling schema evolution and time travel capabilities.
• Design and enforce data quality frameworks, processes, and tools to ensure data accuracy, completeness, and consistency across the enterprise.
• Collaborate with data goverce teams to define policies and ensure compliance with regulatory requirements (e.g., HIPAA, GDPR).
• Implement monitoring and alerting solutions for data pipelines and platform health
• Integrate and configure Starburst (Trino) for federated query capabilities across diverse data sources and BigLake Metastore for unified metadata management
• Provide technical leadership, mentorship, and guidance to data engineers and other technical teams.
• Collaborate closely with business stakeholders, US team and application development teams to understand data needs and deliver solutions.
• Communicate complex technical concepts effectively to both technical and non-technical audiences.
• Stay abreast of emerging technologies and industry trends in data architecture, GCP services, and open-source data platforms
• Provide regular updates on the tasks, status and risks to project manager

Requirements

• Bachelors degree or higher from a reputed university
• 12 to 18 years of progressive experience in data architecture, data engineering or similar role
• 3+ years of hands-on experience designing and implementing large-scale data platforms on Google Cloud Platform (GCP).
• Demonstrated expertise with BigQuery, Dataproc, Dataflow, and other relevant GCP data services.
• Strong proficiency in PySpark for data processing and transformation for both and streaming patterns
• Experience in streaming tools like Kafka, Pub-Sub, spark streaming etc.
• Extensive experience with modern data warehousing principles, data lakes, and Data lakehouse architectures.
• Hands-on experience with open table formats, specifically Apache Iceberg.
• Experience implementing data goverce frameworks, data quality tools, and metadata management solutions.
• Strong understanding of security best practices in a cloud data environment.
• Experience with Infrastructure as Code (e.g., Terraform) and CI/CD pipelines for data solutions is a plus
• Excellent analytical and problem-solving skills with a keen attention to detail
• Strong communication and interpersonal skills, with the ability to explain complex technical concepts to both technical and non-technical audiences

Good to have

• Experience with federated query engines like Starburst (Trino) and unified metadata solutions like BigLake Metastore is highly desirable.
• Proven experience with data transformation tools such as dbt (data build tool).
• Experience in processing streaming data using Kafka / Pub-Sub
• Healthcare industry experience
• Experience in Agile