Sr. Data Engineer

Veriipro

Boston (MA)

On-site

USD 140,000 - 180,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Veriipro is seeking a Lead Data Engineer to design and implement a scalable data ingestion framework using Python and DBT, targeting AWS S3 as the data lake and Snowflake as the data platform.

You will replace legacy ETL with an open-source stack, build reusable connectors, and drive metadata-driven pipelines across multiple source systems. Strong leadership and cloud data experience are essential.

Qualifications

  • 10-12 years of overall experience in data engineering and data integration.
  • Hands-on experience building enterprise-grade data ingestion frameworks.
  • Expert-level Python development and DBT Core expertise.
  • Experience with Snowflake architecture, SQL, and data loading.
  • Strong AWS S3 and cloud-based data integration experience.

Responsibilities

  • Develop a reusable, scalable, technology-agnostic data ingestion framework using Python.
  • Design modular components to support multiple source systems.
  • Develop metadata-driven ingestion capabilities and reusable connectors.
  • Implement full and incremental ingestion patterns including CDC where applicable.
  • Ensure robust error handling, logging, auditing, and retry mechanisms.
  • Build ingestion pipelines to integrate data into AWS S3 and Snowflake.
  • Develop and maintain dbt models for transformations and integration.
  • Implement dbt testing, documentation, and incremental models.
  • Optimize ingestion processes for performance and scalability.
  • Lead with engineering best practices and code quality.
  • Mentor developers and ensure adherence to standards.

Skills

Python
DBT
Data engineering
Snowflake
AWS
SQL
CI/CD
Dagster
Control-M
Docker
Terraform
Apache Iceberg
REST APIs
SFTP
SCRUM

Tools

Snowflake
Dagster
Control-M
Apache Iceberg
Docker
Terraform
AWS S3

Job description

Job Description: Sr. Data Engineer - Data Ingestion Framework

Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

Engagement: Data Platform Modernization / Data Ingestion Framework

Position Overview

We are looking for an experienced Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

Key Responsibilities
1. Data Ingestion Framework Development
  • Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.
  • Design modular and extensible components to support multiple source systems.
  • Develop metadata-driven and configuration-based ingestion capabilities.
  • Build reusable connectors and adapters for different source technologies.
  • Implement full and incremental data ingestion patterns, including CDC where applicable.
  • Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.
2. Data Integration & Lakehouse Architecture
  • Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.
  • Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.
  • Design and implement data lakehouse ingestion patterns using AWS S3.
  • Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.
  • Implement schema handling, data validation, partitioning, and data movement strategies.
3. Python & dbt Engineering
  • Develop production-grade Python applications and reusable ingestion libraries.
  • Build and maintain dbt models for data transformation and integration.
  • Implement dbt testing, documentation, incremental models, and reusable macros.
  • Optimize Python-based ingestion processes and dbt transformations for performance and scalability.
  • Follow engineering best practices for modularity, maintainability, and code quality.
4. Workflow Orchestration
  • Implement and manage workflow orchestration using Control-M and Dagster.
  • Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.
  • Integrate orchestration workflows with Python ingestion components and Snowflake.
5. Testing, Deployment & Production Support
  • Develop unit tests and integration tests for ingestion framework components.
  • Support migration and deployment across DEV, SIT, UAT, and PROD environments.
  • Collaborate with QA teams to perform functional, integration, regression, and performance testing.
  • Troubleshoot ingestion failures, data discrepancies, and performance bottlenecks.
  • Support production deployment, stabilization, documentation, and knowledge transfer.
6. Technical Leadership & Collaboration
  • Work closely with the onshore solution architect, customer stakeholders, and offshore engineering teams.
  • Provide technical guidance, code reviews, and engineering best practices.
  • Participate in architecture discussions, technical design reviews, and sprint planning.
  • Mentor developers and ensure adherence to established engineering standards.
Required Technical Skills
  • Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.
  • DBT: Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.
  • Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.
  • Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.
  • AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.
  • Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.
  • SQL: Advanced SQL development and query optimization skills.
  • CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.
Good-to-Have Skills
  • Hands-on experience withDagster for data orchestration and asset-based pipeline development.
  • Experience with utilizing Control-M JOB scheduler.
  • Experience with Apache Iceberg and open table formats.
  • Experience implementing metadata-driven ingestion frameworks.
  • Experience migrating ETL workloads from SnapLogic or similar integration platforms.
  • Experience with Docker, Terraform, and infrastructure automation.
  • Experience implementing data quality, observability, and monitoring solutions.
  • Experience working in an Agile environment utilizing SCRUM
Qualifications
  • 10-12 years of overall experience in data engineering and data integration.
  • Strong architectural and hands-on engineering capabilities.
  • Experience working on enterprise data modernization initiatives.
  • Excellent communication and stakeholder management skills.
  • Ability to work independently in a distributed onshore/offshore delivery model.
Mandatory Skills
  1. Expert-level Python development.
  2. Strong hands-on dbt experience.
  3. Enterprise data ingestion framework development.
  4. Snowflake and AWS S3 experience.
  5. Strong SQL and data integration experience.
Preferred

Dagster, Control-M, Apache Iceberg, and SnapLogic migration experience.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

firstPRO US • Arlington (VA)

On-site
USD 120,000 - 190,000
Data Engineer
Data Engineer

Tata Consultancy Services Limited • Detroit (MI)

On-site
USD 70,000 - 110,000
Snowflake Data Engineer
Snowflake Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 120,000 - 170,000
Sr. Data Engineer
Sr. Data Engineer

TechDigital Group • Detroit (MI)

On-site
USD 120,000 - 180,000
Snowflake Data Engineer
Snowflake Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

On-site
USD 120,000 - 170,000
Data Engineer
Data Engineer

TDIndustries, Inc. • Dallas (TX)

On-site
USD 108,000 - 132,000
Data Engineer
Data Engineer

firstPRO US • Virginia

On-site
USD 140,000 - 190,000
Senior Data Engineer
Senior Data Engineer

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

Sakata Seed America, Inc. • Woodland (CA)

On-site
USD 110,000 - 125,000
Data Engineer Lead
Data Engineer Lead

VSE AVIATION INC (US) • Phoenix (AZ)

On-site
USD 140,000 - 200,000