Senior Data Engineer : GCP

ORION SYSTEMS; Dadri; Hybrid; INR 1,200,000 - 2,200,000; Full time

Job summary

ORION SYSTEMS in India seeks a senior data engineer to design, build and deploy robust ETL/ELT pipelines on lakehouse platforms such as Google BigQuery or Databricks. You will implement the Bronze-Silver-Gold architecture, ensure data quality, and enable scalable ingestion from multiple sources.

You will optimize performance with partitioning, clustering and Delta Lake, govern data with Dataplex/Unity Catalog, and develop end-to-end workflows with Airflow or Databricks Workflows, collaborating

Qualifications

• Hands-on experience designing ETL/ELT pipelines on lakehouse platforms (BigQuery/Databricks).
• Strong SQL, Python, PySpark, and Spark SQL skills.
• Experience with data governance, security, and data lineage tools.
• Familiarity with Delta Lake, ACID, partitioning and data modelling.

Responsibilities

• Design, build, and deploy robust ETL/ELT pipelines on lakehouse platforms.
• Optimize BigQuery tables and Spark/Delta Lake jobs for performance and cost.
• Implement Bronze-Silver-Gold architecture using Delta Lake/BigQuery datasets.
• Develop and monitor complex data workflows with Airflow/Databricks Workflows.
• Ensure data governance and security using Dataplex/Unity Catalog.

Skills

BigQuery

Databricks

PySpark

Spark SQL

SQL

Python

Airflow

Tools

Delta Lake

BigQuery Data Transfer Service

Pub/Sub

Databricks Auto Loader

Unity Catalog

Dataplex

Job description

Role & responsibilities

• Design, build, and deploy robust ETL/ELT pipelines within the lakehouse platform (Google BigQuery or Databricks) using SQL, Python, PySpark, and Spark SQL.
• Implement and manage the Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake or BigQuery datasets to ensure data quality and progressive data refinement.
• Leverage native ingestion tooling such as BigQuery Data Transfer Service, Pub/Sub streaming, or Databricks Auto Loader for efficient, scalable, and incremental ingestion of data from sources like GA4 into the Bronze layer.
• Develop, schedule, and monitor complex, multi-task data workflows using Cloud Composer (Airflow), BigQuery scheduled queries, or Databricks Workflows.
• Optimize BigQuery tables (partitioning, clustering, materialised views) and Spark jobs / Delta Lake tables (using techniques like OPTIMIZE, Z-ORDER, and partitioning) for high performance and cost efficiency.
• Implement data governance, security, and discovery using Dataplex / BigQuery policy tags or Unity Catalog, including managing access controls and data lineage.
• Write complex, customized SQL queries to manipulate data and support ad-hoc analytical requests from business teams.
• Develop strategies for data ingestion from multiple sources, using various techniques including streaming, API consumption, and replication.
• Document data engineering processes, data models, and technical specifications for

the lakehouse platform.

• Conform to agile development practices, including version control (Git), continuous

integration/delivery (CI/CD), and test-driven development.

• Provide production support for data pipelines, actively monitoring and resolving

issues to ensure the continuous flow of critical data.

• Collaborate with analytics and business teams to understand data requirements and

deliver well-modelled, performant datasets in the gold layer.

Preferred candidate profile

• Lakehouse Platform Expertise (Google BigQuery and/or Databricks):
• BigQuery: Deep, hands-on experience with BigQuery architecture, including partitioning, clustering, materialised views, slot/cost optimisation, and diagnosing query performance using query plans and INFORMATION_SCHEMA.
• Apache Spark / Delta Lake: Strong experience with Spark architecture, writing and optimising PySpark and Spark SQL jobs, and building reliable pipelines on Delta Lake.
• Proficient with ACID transactions, time travel, schema evolution, and DML operations (MERGE, UPDATE, DELETE).
• Data Ingestion: Experience with modern ingestion tools, such as BigQuery Data Transfer Service, Pub/Sub / Dataflow streaming, Databricks Auto Loader, and COPY INTO for scalable file processing.
• Data Governance: Strong understanding of data governance concepts and practical experience implementing security, lineage, and discovery using Dataplex, BigQuery IAM and policy tags, or Unity Catalog.