Lead II - Data Engineering

UST; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

UST is seeking a Sr Data Platform Engineer in Bengaluru to lead tenant consolidation, unify data catalogs, and align security across cloud platforms. The role involves migrating pipelines, refactoring Spark workflows, and optimizing costs while maintaining governance and cross-account access.

You will collaborate with data engineers and product teams to deliver a scalable, unified data platform. Candidates should have 4–6 years of experience in data platform engineering or architecture, strong

Qualifications

• 4-6 years of experience in Data Platform Engineering or Data Architecture.
• Experience with data platform migrations and cloud account consolidations.
• Ability to act as technical lead, mapping migration schedules and coordinating tasks with mid-level engineers.
• Proficiency in Python and SQL, and IaC tooling like Terraform.

Responsibilities

• Contribute to architectural design for tenant consolidation and unified platform.
• Unify data catalogs and governance across Databricks or Snowflake environments.
• Migrate legacy ETL/ELT pipelines and Spark workflows to the new platform.
• Audit and align security models, IAM roles, and cross-account access.

Skills

AWS

Data Engineering

Databricks

Python

Job description

Role Description

Sr Data Platform Engineer Tenant Consolidation: Contribute to the architectural design and execution strategy to merge separate Databricks workspaces or Snowflake accounts into a unified platform. Metadata & Catalog Unification: Consolidate data catalogs, aligning separate Unity Catalogs or Snowflake Shares into a single, company-wide governance model. Security & Access Alignment: Audit and merge distinct security models, unifying IAM roles, active directory syncs, and row/column-level access controls. Pipeline & Code Migration: Refactor and migrate legacy ETL/ELT pipelines, Spark workflows, and orchestration routines to point to the new unified tenant. Cost & Asset Optimization: Identify overlapping data assets, compute resources, and storage tiers during the merge to optimize cloud spend. Technical Requirements Cloud Infrastructure: Proven experience managing AWS ecosystems (S3, IAM, CloudTrail, VPC, Route53) with an emphasis on cross-account access. Data Platform Mastery: Deep expertise in Databricks (workspace administration, Delta Lake, Photon) OR Snowflake (multi-account management, Snowpark, Virtual Warehouses). Infrastructure as Code (IaC): Strong experience writing Terraform scripts to provision and replicate workspace configurations. Development Skills: Python and SQL skills, with experience auditing and refactoring code written by other teams. Experience and Qualifications 4-6 years of experience in Data Platform Engineering, or Data Architecture. good to have Prior experience working on data platform migrations, cloud account consolidations, Ability to act as a technical lead, mapping out migration schedules and coordinating tasks with the mid-level engineer.

Role Description

Sr Data Platform Engineer Tenant Consolidation: Contribute to the architectural design and execution strategy to merge separate Databricks workspaces or Snowflake accounts into a unified platform. Metadata & Catalog Unification: Consolidate data catalogs, aligning separate Unity Catalogs or Snowflake Shares into a single, company-wide governance model. Security & Access Alignment: Audit and merge distinct security models, unifying IAM roles, active directory syncs, and row/column-level access controls. Pipeline & Code Migration: Refactor and migrate legacy ETL/ELT pipelines, Spark workflows, and orchestration routines to point to the new unified tenant. Cost & Asset Optimization: Identify overlapping data assets, compute resources, and storage tiers during the merge to optimize cloud spend. Technical Requirements Cloud Infrastructure: Proven experience managing AWS ecosystems (S3, IAM, CloudTrail, VPC, Route53) with an emphasis on cross-account access. Data Platform Mastery: Deep expertise in Databricks (workspace administration, Delta Lake, Photon) OR Snowflake (multi-account management, Snowpark, Virtual Warehouses). Infrastructure as Code (IaC): Strong experience writing Terraform scripts to provision and replicate workspace configurations. Development Skills: Python and SQL skills, with experience auditing and refactoring code written by other teams. Experience and Qualifications 4-6 years of experience in Data Platform Engineering, or Data Architecture. good to have Prior experience working on data platform migrations, cloud account consolidations, Ability to act as a technical lead, mapping out migration schedules and coordinating tasks with the mid-level engineer.

Skills

AWS, Data Engineering, Databricks, Python