Data Engineer

Siri InfoSolutions Inc; Raleigh (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Siri InfoSolutions Inc in Raleigh, NC is seeking a Data Engineer with 8+ years to own data pipeline architecture and deliver scalable Snowflake-based solutions across ingestion, transformation, and governance.

You will build Python ingestion frameworks, DBT models, and orchestrate workflows with Astronomer Airflow, GitLab, and CI/CD, while ensuring schema drift detection, controlled changes, and secure data masking.

Qualifications

• 8+ years of hands-on data engineering experience.
• Deep expertise with Snowflake, including data masking policies, RBAC, performance tuning, and advanced SQL.
• Strong experience with Qlik Replicate for CDC and database replication.
• Excellent proficiency in Python and Pyspark for ingestion frameworks and automation.
• Hands on experience with DBT Cloud and Astronomer Airflow.
• Experience with schema drift detection and schema evolution patterns.
• Experience with GitLab and CI/CD pipelines.
• Familiarity with Protegrity or similar data protection platforms.

Responsibilities

• Data Pipeline Architecture & Development.
• Design and implement scalable, resilient data pipelines using Snowflake features.
• Build and maintain DBT models with testing, documentation, and lineage.
• Develop Python ingestion frameworks for files and APIs, including schema validation and retries.
• Ingest CSV, fixed width, JSON, XML, Excel, and semi-structured formats.
• Coordinate schema changes through CI/CD workflows.
• Configure and manage Qlik Replicate tasks for CDC and full load replication.
• Implement and enforce data governance, security, and tokenization policies.
• Build and schedule workflows using Astronomer Airflow and integrate with GitLab/CI/CD.

Skills

Snowflake

Python

Pyspark

DBT Cloud

Astronomer Airflow

Qlik Replicate

GitLab

CI/CD

Protegrity

RBAC

Snowpipe

Streams

Dynamic Tables

SQL

Schema drift

Schema evolution

Oracle

SQL Server

DB2

Data Masking

Tools

Snowflake

Qlik Replicate

DBT Cloud

Astronomer Airflow

GitLab

Azure DevOps

Protegrity

Job description

Role: Data Engineer

• * Data Engineer with 8+ experience

Descriptions:

" Responsibilities:

• Data Pipeline Architecture & Development
• Design and implement scalable, resilient data pipelines using Snowflake features
• including Snowpipe, Tasks, Streams, Dynamic Tables, and advanced SQL.
• Build and maintain DBT models with strong testing, documentation, and lineage.
• Develop Python ingestion frameworks for files and APIs, including schema validation,
• retries, and metadata capture.
• Engineer ingestion for CSV, fixed width multi record layouts, JSON, XML, Excel,
• and semi structured formats.
• Design Mainframe VSAM data ingestion pattern for complex EBCDIC data formats.
• Schema Drift & Schema Evolution
• Detect, analyze, and manage schema drift across file, API, and replicated database sources.
• Implement metadata driven schema evolution strategies to ensure downstream stability.
• Coordinate schema changes through controlled CI/CD workflows.
• Database Replication & CDC
• Configure and manage Qlik Replicate tasks for CDC and full load replication from Oracle,
• SQL Server, and DB2.
• Ensure idempotent, auditable, and recoverable replication pipelines with strong
• monitoring and reconciliation.
• Data Governance, Security & Tokenization
• Implement and maintain Snowflake Data Masking policies, including dynamic
• masking, conditional masking, and role based masking rules.
• Apply Protegrity tokenization for sensitive data fields across ingestion and transformation
• layers.
• Enforce RBAC, data access controls, and governance standards across Snowflake and
• supporting systems.
• Orchestration & Automation
• Build and schedule workflows using Astronomer Airflow, ensuring dependency
• management, retries, SLAs, and observability.
• Integrate pipelines with enterprise DevOps processes using GitLab and Azure DevOps
• for CI/CD automation.
• Version Control & Code Quality
• Manage code repositories using GitLab, including branching strategies, merge requests,
• code reviews, and approvals.
• Monitoring, Alerting & Performance Optimization
• Implement monitoring and alerting for ingestion pipelines, schema drift, replication, and
• transformation workloads.
• Optimize Snowflake compute, storage, and query performance; scale ingestion pipelines
• to meet evolving data volume and latency requirements.

" 8+ years of hands on data engineering experience.

• Deep expertise with Snowflake, including data masking policies, RBAC, performance tuning, and advanced SQL.
• Strong experience with Qlik Replicate for CDC and database replication.
• Excellent proficiency in Python and Pyspark for ingestion frameworks and automation.
• Hands on experience with DBT Cloud and Astronomer Airflow.
• Experience with schema drift detection and schema evolution patterns.
• Experience with GitLab and CI/CD pipelines.
• Familiarity with Protegrity or similar data protection platforms.