Key Responsibilities
- Lakehouse Architecture & Development: Design, deploy, and maintain a scalable GCP-based Lakehouse ecosystem leveraging Apache Iceberg for transactional consistency, time-travel, and efficient storage management.
- Pipeline Engineering: Architect, build, and optimize high-throughput ETL/ELT data pipelines using Python and Apache Airflow for orchestrating complex dependency workflows.
- Big Data Processing: Implement large-scale distributed data processing workloads and transformation pipelines using Apache Spark.
- Data Warehousing & Analytics: Leverage BigQuery for high-performance analytical queries, cost optimization, partition/cluster management, and serving structured data to downstream consumers.
- Transformation & Quality: Write clean, modular, and testable transformation code using dbt (Data Build Tool), embedding automated data testing, documentation,and lineage tracking.
- Data Modeling: Lead dimensional and semantic data modeling efforts (Star/Snowflake schemas, One-Big-Table, and Lakehouse medallion architectures) to ensure high query performance and seamless business usability.
- Performance Tuning & SQL Mastery: Write, optimize, and troubleshoot complex SQL queries to ensure low latency and cost-efficient processing across large datasets.
- Collaboration: Partner closely with data scientists, analytics teams, and product managers to understand data requirements and deliver robust, production-ready data products.
3+ years of hands-on experience designing and implementing enterprise-grade data platforms and pipelines.
- Cloud Platform: Proven production experience with Google Cloud Platform (GCP) core data services (BigQuery, Cloud Storage, Dataflow/Dataproc, IAM).
- Lakehouse & Table Formats: Deep understanding of modern lakehouse concepts and hands-on experience with Apache Iceberg.
- Transformation & Orchestration: Advanced proficiency in dbt for pipeline transformations and Apache Airflow for workflow orchestration.
- Big Data & Compute: Strong working knowledge of Apache Spark for distributed data processing.
Programming & Querying
- Advanced Python programming skills for data engineering tasks and custom operator development.
- Expert-level SQL writing and performance tuning capabilities.
Data Modeling: Demonstrated expertise in dimensional modeling, data warehousing concepts, and schema design for analytical workloads.
Methodologies: Strong grasp of modern ETL/ELT design patterns, CI/CD for data pipelines, data governance, and data quality best practices.