Data Architect with GCP

Pyramid Technology Solutions; Dallas (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Pyramid Technology Solutions in Dallas seeks an experienced data architecture lead to define and guide the implementation of a metadata-driven ETL framework supporting databases, files, APIs, and cloud sources. You will architect reusable metadata models, oversee MVC-based orchestration, and drive CI/CD automation with GitHub Actions, JFrog, Harness, and Liquibase while optimizing BigQuery performance.

You will collaborate with stakeholders to finalize architecture decisions, document standards,

Qualifications

• Experience designing metadata-driven ETL frameworks.
• Proven ability to architect scalable data pipelines.
• Strong knowledge of data lake architectures and governance.

Responsibilities

• Define and guide implementation of a metadata-driven ETL framework.
• Architect reusable metadata models for config, pipelines, and audits.
• Provide MVC-based orchestration direction and standardized contracts.
• Design scalable data ingestion patterns (full/incremental/CDC/batch).
• Oversee GCP/BigQuery architecture and secure connectivity.
• Review CI/CD automation using GitHub Actions, JFrog, Harness, Liquibase, Helm.
• Guide Python, PySpark, SQL, and performance tuning for production systems.
• Support UAT with validation scenarios and sign-off criteria.
• Collaborate with stakeholders to finalize architecture docs and roadmaps.

Skills

Python

PySpark

SQL

BigQuery

PostgreSQL

Data modeling

Performance tuning

Production-ready framework design

Tools

GitHub Actions

JFrog

Harness

Liquibase

Helm

Kubernetes (OCP)

Job description

• Define and guide implementation of a metadata-driven generic ETL framework supporting multiple source types including databases, files, APIs, MongoDB, Splunk, VDS, and Active Directory.


• Architect reusable metadata models using MD_ static metadata tables and OPS_ operational/runtime tables for source configuration, pipeline definition, run tracking, audit logging, checkpointing, schema drift, and notifications.


• Provide architecture direction for MVC-based orchestration, including repository layer, service layer, wrapper, controller, execution plan models, and standardized request/response contracts.


• Design scalable ingestion patterns for full load, incremental load, watermark-based processing, CDC-style patterns, batch ingestion, API ingestion, and file-based ingestion.


• Provide technical leadership for GCP / BigQuery architecture, including S3 to BigQuery integration, schema mapping, dataset design, partitioning, clustering, performance optimization, and secure connectivity.


• Review and guide CI/CD automation using GitHub Actions, JFrog, Harness, Liquibase, Helm, OCP deployments, and environment-specific configuration management.


• Provide technical guidance to team on Python, PySpark, SQL, BigQuery, PostgreSQL, data modeling, performance tuning, and production ready framework design.


• Support UAT and production readiness by defining validation scenarios, reconciliation strategy, performance testing approach, security validation, runbooks, and sign-off criteria.


• Partner with client stakeholders to finalize architecture decisions, design documents, Confluence documentation, implementation standards, and technical roadmaps.


• Ensure the solution is secure, scalable, maintainable, metadata-driven, auditable, and operationalized.

Data Lake Architecture & Storage

Data Ingestion & Orchestration

Data Processing & Transformation

DevOps / CI-CD

• Experience implementing CI/CD pipelines for data and analytics workloads


• Familiarity with Git-based source control, build automation, and deployment strategies