Lead Data Engineer - GCP

Elabs Infotech; India; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Elabs Infotech is seeking a Data Engineer with deep expertise in Python, SQL, and GCP to design, build, and optimize data pipelines. The role focuses on handling XML/JSON payloads, creating robust data models in BigQuery, and integrating with Infor ION APIs.

You will implement end-to-end ETL/ELT workflows, ensure data quality and security, and collaborate with cross-functional teams to meet business needs on the GCP platform.

Qualifications

• Expert proficiency in Python and SQL.
• Experience parsing and flattening complex XML and JSON payloads.
• Strong knowledge of Google Cloud Platform (GCP) including BigQuery schema design, GCS, and IAM.
• Experience with GCP data ingestion services such as BigQuery Data Transfer Service and Cloud Data Fusion.

Responsibilities

• Lead design and development of scalable data pipelines on GCP.
• Build end-to-end ETL/ELT workflows and automate data integration.
• Manage and optimize BigQuery schemas for performance and cost.
• Ingest data from multiple sources including Infor ION API.
• Ensure security best practices with IAM roles and permissions.
• Collaborate with Data Engineers, Analysts, and stakeholders to ensure data quality.

Skills

Python

SQL

XML

JSON

GCP

ETL/ELT

Tools

BigQuery

GCS

IAM

BigQuery Data Transfer Service

Cloud Data Fusion

Infor ION API

Job description

Role & responsibilities

• Expert proficiency in Python and SQL.
• Experience in parsing and flattening complex XML and JSON payloads.
• Strong knowledge of the Google Cloud Platform (GCP) ecosystem, including BigQuery schema design, Google Cloud Storage (GCS), and Identity and Access Management (IAM).
• Familiarity with GCP data ingestion services and patterns, such as BigQuery Data Transfer Service and Cloud Data Fusion.
• Hands-on experience in building end-to-end data pipelines and automating ETL/ELT workflows.
• Experience integrating with Infor ION API endpoints.
• Ability to design, deploy, and manage data warehouse architectures on Google Cloud Platform (GCP).

Preferred candidate profile

• Lead the design and development of scalable data pipelines and data architectures on Google Cloud Platform (GCP).
• Develop, maintain, and optimize ETL/ELT processes to automate data integration and workflow execution.
• Design and optimize BigQuery schemas to ensure high performance, scalability, and cost efficiency.
• Manage data ingestion from multiple sources, including Infor ION API endpoints.
• Implement and enforce security best practices using Identity and Access Management (IAM) roles and permissions.
• Collaborate with cross-functional teams, including Data Engineers, Analysts, and Business Stakeholders, to ensure data quality, integrity, and availability.
• Monitor, troubleshoot, and optimize data pipeline performance, reliability, and overall system health.