Data Architect & Engineer: Medallion Lakehouse & dbt Expert

BTI Solutions, Inc.; Plano (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

BTI Solutions, Inc. seeks a skilled Data Engineer to design and implement end-to-end data pipelines.

You will ingest data from SQL Server and Oracle into the Bronze layer using CDC-based incremental ingestion, and build dbt models for the Silver layer including cleansing, standardization, and quality tests. You will migrate Alteryx logic to Gold layer, configure Iceberg tables and a REST Catalog, and develop Airflow DAGs to manage dependencies, scheduling, and failures.

Qualifications

• 3+ years of hands-on data engineering experience in production environments.
• Advanced SQL skills, including window functions, CTEs, and query tuning.
• Hands-on dbt experience, including model design, incremental strategies, macros, and tests.
• Python proficiency for data processing and automation, including libraries such as pandas.
• Experience developing Apache Airflow DAGs.
• Experience with at least one analytical engine: Trino/Presto, Apache Spark, or DuckDB.
• Experience with at least one lakehouse table format: Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with Medallion Architecture and star schema/dimensional modeling.
• Experience integrating relational database sources, such as Microsoft SQL Server and Oracle, using CDC or batch processing

Responsibilities

• Ingest data from sources such as MS SQL Server and Oracle into Bronze with CDC-based ingestion.
• Develop dbt models for the Silver layer including cleansing, standardization, deduplication, and data quality testing.
• Develop Gold layer data marts and migrate existing Alteryx transformation logic.
• Configure Iceberg tables and REST Catalog with schema evolution and time travel.
• Develop and operate Airflow DAGs with dependency management, scheduling, and failure notifications.
• Develop an MCP server enabling AI agents to query Gold-layer data marts and access catalog/metadata.
• Manage metadata and data lineage using OpenMetadata.
• Automate Excel reporting using Python and openpyxl.

Skills

SQL

Window functions

CTEs

Python

pandas

dbt

Airflow

Spark/Presto/DuckDB

Iceberg/Delta/Hudi

Kubernetes

MinIO

Tools

dbt

Airflow

Apache Iceberg

OpenMetadata

Job description

BTI Solutions, Inc. seeks a skilled Data Engineer to design and implement end-to-end data pipelines.

You will ingest data from SQL Server and Oracle into the Bronze layer using CDC-based incremental ingestion, and build dbt models for the Silver layer including cleansing, standardization, and quality tests. You will migrate Alteryx logic to Gold layer, configure Iceberg tables and a REST Catalog, and develop Airflow DAGs to manage dependencies, scheduling, and failures.