Lead I - Data Engineering (Databricks (workspace administration, Delta Lake, Photon)

UST; Bengaluru; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

UST in Bengaluru is seeking a Data Platform Engineer for Merger & Migration to execute hands-on data asset migrations, update pipelines, and set up unified workspaces using Terraform.

The role requires 2-4 years in Data Engineering or Cloud Operations, with hands-on experience in Snowflake or AWS, Python and SQL, and CI/CD practices to ensure zero data loss during cutovers.

Qualifications

• 2-4 years of experience in Data Engineering, Platform Operations, or Cloud Operations.
• Strong troubleshooting and collaboration skills to resolve migration roadblocks.
• Proficiency with Terraform for provisioning infrastructure and Git/CI-CD workflows.

Responsibilities

• Hands-on migration of data assets, schemas, tables, and notebooks from source to target environments.
• Update and test data pipelines, adjusting storage paths, API endpoints, and connection strings to the unified architecture.
• Spin up unified workspaces, clusters, and warehouses using Terraform and platform APIs in line with company standards.
• Recreate and map user groups, access permissions, and roles to the central governance model.
• Run data validation tests and monitor migration logs to ensure zero data loss or downtime during cutovers.

Skills

Snowflake

AWS

AWS IAM

RBAC

SQL

Data Validation

Production Support

CI/CD

Job description

Role Description

Data Platform Engineer Merger & Migration Responsibilities Hands-on Migration: Execute the moving of data assets, schemas, tables, and notebooks from the source tenant to the target environment. Pipeline Updates: Update and test data pipelines, changing storage paths, API endpoints, and connection strings to match the new unified architecture. Workspace Setup: Use Terraform and platform APIs to spin up new, unified workspaces, clusters, and warehouses according to the new company standards. Permissions Mapping: Recreate and map user groups, access permissions, and roles from the old systems into the new centralized governance model. Monitoring & Validation: Run data validation tests and monitor migration logs to ensure zero data loss or downtime during cutovers. Technical Requirements Cloud Foundations: Solid hands-on experience using AWS (S3, IAM, EC2, Lambda) in a multi-account setup. Platform Experience: Practical administration experience with Databricks (clusters, notebooks, jobs) OR Snowflake (warehouses, roles, storage integration). Automation Tools: Comfortable using Terraform for provisioning infrastructure and Git/CI/CD for deployment pipelines. Core Coding: Proficient in Python and SQL for writing tests, scripting automation tasks, and verifying data structures. Experience and Qualifications 2-4 years of experience working in Data Engineering, Platform Operations, or Cloud Operations. Strong troubleshooting skills and the ability to work collaboratively to solve migration roadblocks.

Data Platform Engineer Merger & Migration Responsibilities Hands-on Migration: Execute the moving of data assets, schemas, tables, and notebooks from the source tenant to the target environment. Pipeline Updates: Update and test data pipelines, changing storage paths, API endpoints, and connection strings to match the new unified architecture. Workspace Setup: Use Terraform and platform APIs to spin up new, unified workspaces, clusters, and warehouses according to the new company standards. Permissions Mapping: Recreate and map user groups, access permissions, and roles from the old systems into the new centralized governance model. Monitoring & Validation: Run data validation tests and monitor migration logs to ensure zero data loss or downtime during cutovers. Technical Requirements Cloud Foundations: Solid hands-on experience using AWS (S3, IAM, EC2, Lambda) in a multi-account setup. Platform Experience: Practical administration experience with Databricks (clusters, notebooks, jobs) OR Snowflake (warehouses, roles, storage integration). Automation Tools: Comfortable using Terraform for provisioning infrastructure and Git/CI/CD for deployment pipelines. Core Coding: Proficient in Python and SQL for writing tests, scripting automation tasks, and verifying data structures. Experience and Qualifications 2-4 years of experience working in Data Engineering, Platform Operations, or Cloud Operations. Strong troubleshooting skills and the ability to work collaboratively to solve migration roadblocks.

Skills

Snowflake, AWS, AWS IAM, RBAC, SQL, Data Validation, Production Support, CI/CD