Senior Data Engineer Offshore

Photon; Thiruvallur District; Hybrid; INR 2,000,000 - 4,000,000; Full time

Benefits offered by this job

Collibra

Job summary

Photon is seeking an experienced data engineer to design and optimize data pipelines across SQL Server, Snowflake, and ODS environments. You will build and maintain CI/CD pipelines for Snowflake deployments, implement data quality checks, and migrate batch flows to API-based architectures.

Ideal candidates have 7+ years in data engineering with SQL, Python, and strong Snowflake skills, plus experience with dbt, Informatica, and DevOps practices. Knowledge of finance domain is a plus.

Qualifications

• 7+ years in data engineering with pipeline development and data modeling.
• Strong SQL proficiency (SQL Server, Postgres) and Python.
• Proficiency in Snowflake schema design and loading patterns.
• Experience building DevOps/DataOps pipelines with Git and CI/CD.
• Hands-on with dbt, Schemachange, Flyway for migrations.
• ETL/ELT tooling experience; Informatica preferred.
• Understanding of ODS concepts and dimensional modeling.
• Experience integrating REST/API data sources during modernization.
• Experience using Generative AI tools to enhance coding and testing.
• Background in financial services or lending domain.

Responsibilities

• Assess existing SQL Server (Atlas) and Snowflake environments to identify gaps and inefficiencies.
• Build and maintain data pipelines supporting the ODS, including ingestion, transformation, and loading of key entities.
• Implement data dictionary standards and entity models as per Data Architect guidelines.
• Design, implement, and maintain CI/CD pipelines to automate Snowflake deployments.
• Manage schema migrations using dbt, Schemachange, Flyway, or Terraform.
• Execute batch-to-API migration, re-engineering Informatica/EDW flows to FDR APIs.
• Implement automated data quality frameworks and parity checks between legacy and new ODS.
• Collaborate with domain teams to engineer pipelines for customer, loan, payment data.
• Apply Snowflake best practices for schemas, compute, and workload separation.
• Partner with data architect and downstream consumers to ensure data accuracy and observability.
• Leverage Generative AI tools to accelerate development and write tests.

Skills

SQL

Python

Snowflake

DevOps/DataOps

REST APIs

Generative AI

Financial services domain

Dimensional modeling

Data dictionaries

CI/CD

Education

Tools

dbt

Schemachange

Flyway

Informatica

Git

CI/CD tools

Airflow

Prefect

Dagster

Collibra

Alation

Job description

Job Description:

Key Responsibilities

• Assess existing SQL Server (Atlas) and Snowflake environments to identify pipeline gaps, data quality issues, and structural inefficiencies
• Build and maintain data pipelines supporting the Operational Data Store (ODS), including ingestion, transformation, and loading of key business entities
• Implement, version-control, and enforce data dictionary standards, logical/physical entity models, and naming conventions established by the Data Architect
• Design, implement, and maintain CI/CD pipelines (e.g., GitHub Actions, GitLab CI) to automate the deployment of Snowflake schemas, database structures, and pipeline code
• Manage database schema migrations as code using tools such as dbt, Schemachange, Flyway, or Terraform
• Execute the batch-to-API migration roadmap — re-engineering Informatica/EDW batch flows to consume FDR APIs
• Implement automated data quality frameworks, testing validations (e.g., dbt tests, Great Expectations), and data parity reconciliation routines between legacy and new ODS environments
• Collaborate with domain teams to engineer pipelines for customer, loan, payment, and interaction data entities
• Apply Snowflake best practices in structuring schemas, managing compute, and separating analytical from operational workloads
• Partner with the internal data architect and downstream consumers to ensure data accuracy, lineage, and observability across the platform
• Leverage generative AI tools (e.g., coding assistants, LLMs) to accelerate pipeline development, optimize query performance, write documentation, and generate automated tests

Must-Have Skills

• 7+ years in data engineering roles with a strong focus on pipeline development, CI/CD setup, and database modeling
• Strong programming skills in SQL (Sql Server and Postgres) and Python for data engineering, automation, and pipeline development
• Proficiency in Snowflake — including schema design, performance tuning, and data loading patterns
• Hands-on experience building DevOps/DataOps pipelines (Git, CI/CD tools, automated deployment strategies)
• Hands-on experience with database migration tools / schema management systems (e.g., dbt, Schemachange, Flyway)
• Hands-on experience with ETL/ELT tools and batch processing (Informatica experience strongly preferred)
• Solid understanding of ODS concepts, dimensional modeling, and managing data dictionaries
• Experience integrating with REST/API-based data sources as part of modernization or migration efforts
• Demonstrated experience using Generative AI coding assistants (e.g., Copilot, ChatGPT, Gemini) to enhance coding productivity, write tests, and troubleshoot complex SQL/Python scripts
• Background in financial services or lending domain

Nice to Have

• Experience with modern orchestrators (e.g., Apache Airflow, Prefect, Dagster) or Snowflake-native orchestration (Tasks, Dynamic Tables)
• Familiarity with deploying semantic layers or metadata-driven catalog systems (Collibra, Alation) to support downstream AI/ML and Natural Language query agents
• Experience working with FDR or similar loan servicing platforms
• Familiarity with data mesh principles and domain-oriented data ownership
• Exposure to AI/ML data pipeline requirements and feature engineering
• Experience with data governance and cataloguing tools (Collibra, Alation, etc.)
• Background in student lending or consumer finance

Requirements: