Senior Data Pipeline Architect - Snowflake & Airflow

Tata Consultancy Services; Naperville (IL); On-site; USD 115,000 - 125,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Job summary

Tata Consultancy Services in Naperville, IL, is seeking a Senior Data Engineer to architect and optimize enterprise-grade data pipelines across cloud, on-premises, and hybrid environments. You will leverage Snowflake, Qlik Replicate, DBT Cloud, and Astronomer Airflow to build scalable pipelines, implement schema governance, and drive CI/CD automation with a focus on security and governance.

This role demands strong engineering discipline and a proactive approach to performance tuning and

Qualifications

• Experience designing scalable data pipelines on Snowflake and cloud platforms.
• Proficiency with Python-based ingestion frameworks and DBT modeling.
• Experience with CI/CD workflows and metadata-driven design.

Responsibilities

• Data Pipeline Architecture & Development
• Design and implement scalable, resilient data pipelines using Snowflake features including Snowpipe, Tasks, Streams, Dynamic Tables, and advanced SQL.
• Build and maintain DBT models with strong testing, documentation, and lineage.
• Develop Python ingestion frameworks for files and APIs, including schema validation, retries, and metadata capture.
• Engineer ingestion for CSV, fixed width multi record layouts, JSON, XML, Excel, and semi structured formats.
• Design Mainframe VSAM data ingestion pattern for complex EBCDIC data formats.
• Schema Drift & Schema Evolution
• Detect, analyze, and manage schema drift across file, API, and replicated database sources.
• Implement metadata driven schema evolution strategies to ensure downstream stability.
• Coordinate schema changes through controlled CI/CD workflows.
• Database Replication & CDC
• Configure and manage Qlik Replicate tasks for CDC and full load replication from Oracle, SQL Server, and DB2.
• Ensure idempotent, auditable, and recoverable replication pipelines with strong monitoring and reconciliation.
• Data Governance, Security & Tokenization
• Implement and maintain Snowflake Data Masking policies, including dynamic masking, conditional masking, and role based masking rules.
• Apply Protegrity tokenization for sensitive data fields across ingestion and transformation
• Enforce RBAC, data access controls, and governance standards across Snowflake and supporting systems.
• Orchestration & Automation
• Build and schedule workflows using Astronomer Airflow, ensuring dependency management, retries, SLAs, and observability.
• Integrate pipelines with enterprise DevOps processes using GitLab and Azure DevOps for CI/CD automation. Version Control & Code Quality
• Manage code repositories using GitLab, including branching strategies, merge requests, code reviews, and approvals.
• Monitoring, Alerting & Performance Optimization
• Implement monitoring and alerting for ingestion pipelines, schema drift, replication, and transformation workloads.
• Optimize Snowflake compute, storage, and query performance; scale ingestion pipelines to meet evolving data volume and latency requirements.

Skills

Data architecture

Python scripting

SQL optimization

Tools

Snowflake

Qlik Replicate

DBT Cloud

Astronomer Airflow

Python

Pyspark

GitLab

CI/CD

Job description

Tata Consultancy Services in Naperville, IL, is seeking a Senior Data Engineer to architect and optimize enterprise-grade data pipelines across cloud, on-premises, and hybrid environments. You will leverage Snowflake, Qlik Replicate, DBT Cloud, and Astronomer Airflow to build scalable pipelines, implement schema governance, and drive CI/CD automation with a focus on security and governance.

This role demands strong engineering discipline and a proactive approach to performance tuning and