Senior Data Engineer

BrickRed Systems LLC; Seattle (WA); On-site; USD 150,000 - 230,000; Full time

Job summary

BrickRed Systems is seeking a Senior Data Engineer to design, develop, and optimize enterprise-scale data pipelines and platforms supporting Finance and Data Intelligence initiatives. You will leverage Snowflake, Databricks, Delta Lake, DLT, Unity Catalog, dbt, and PySpark to deliver scalable, secure data solutions.

You will build ELT/ETL pipelines for billing, GL, OPEX, and more, implement real-time processing with Kafka and Spark, and lead platform standards, CI/CD, and governance across Azure

Qualifications

• 7+ years experience with Snowflake and Delta Lake.
• 7+ years experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years experience with dbt and PySpark.
• Strong SQL and Python for data pipeline development.
• Experience with ETL/ELT, incremental ingestion, CDC and fault-tolerant processing.
• Knowledge of Azure or AWS cloud data platforms.
• Experience with data quality, testing, CI/CD, security, and governance.

Responsibilities

• Design and develop scalable ELT/ETL pipelines for finance and revenue data.
• Build batch, incremental, and streaming ingestion pipelines with CDC and event-driven patterns.
• Ensure pipeline reliability with idempotent design, error handling, and retries.
• Lead development and optimization of Snowflake and Databricks platforms (Snowpipe, streams, tasks, DLT, Unity Catalog).
• Develop modular data transformations using dbt and PySpark.
• Implement cloud-native data platforms on Azure or AWS with IaC and cost optimization.
• Develop real-time processing solutions with Kafka, Event Hub, and Spark Structured Streaming.
• Establish data quality, validation, monitoring, testing, and observability standards.
• Implement CI/CD, version control, and automated deployment workflows.
• Ensure data security, governance, lineage, auditability, and RBAC/PII controls.
• Collaborate with architects, product managers, analysts, and stakeholders to deliver reliable data solutions.
• Provide technical leadership, mentor engineers, and drive platform standards.

Skills

Snowflake

Delta Lake

Databricks

Delta Live Tables

Unity Catalog

dbt

PySpark

SQL

Python

Data modeling

ETL/ELT

CDC

Cloud platforms

CI/CD

Security & governance

Tools

Airflow

Azure Data Factory

Terraform

Job description

We are seeking a Senior Data Engineer to design, develop, and optimize enterprise-scale data pipelines and platforms supporting Finance and Data Intelligence initiatives. The ideal candidate will have strong expertise in Snowflake, Databricks, Delta Lake, Delta Live Tables (DLT), Unity Catalog, dbt, and PySpark, along with experience building reliable, scalable, and secure data solutions.

Key Responsibilities

• Design and develop scalable ELT/ETL pipelines for finance and revenue data, including billing, general ledger (GL), and operating expenses (OPEX).
• Build batch, incremental, and streaming ingestion pipelines using CDC, watermarking, and event-driven patterns.
• Ensure pipeline reliability through idempotent design, error handling, retries, and fault-tolerance mechanisms.
• Lead development and optimization of Snowflake and Databricks platforms, including Snowpipe, streams, tasks, Delta Live Tables, and Unity Catalog.
• Develop modular, tested, and reusable data transformations using dbt and PySpark.
• Implement cloud-native data platforms using Azure or AWS, including infrastructure-as-code and cost optimization.
• Develop real-time processing solutions using Kafka, Event Hub, and Spark Structured Streaming.
• Establish data quality, validation, monitoring, testing, and observability standards.
• Implement CI/CD practices, version control, and automated deployment workflows.
• Ensure data security, governance, lineage, auditability, and compliance, including RBAC and PII/CPNI controls.
• Collaborate with architects, product managers, analysts, and business stakeholders to deliver reliable data solutions.
• Provide technical leadership, mentor engineers, and drive platform standards and continuous improvement.

Required Qualifications

• 7+ years of experience with Snowflake and Delta Lake.
• 7+ years of experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• 5+ years of experience with dbt and PySpark.
• Strong expertise in SQL, Python, data modeling, and enterprise-scale data pipeline development.
• Experience with ETL/ELT, incremental ingestion, CDC, pipeline optimization, and fault-tolerant processing.
• Knowledge of cloud data platforms, preferably Azure or AWS.
• Experience with data quality, testing, CI/CD, security, and data governance.
• Strong problem-solving, communication, and technical leadership skills.

Preferred Qualifications

• Experience with Airflow or Azure Data Factory.
• Familiarity with Terraform or Bicep for infrastructure provisioning.
• Experience with Kafka, Event Hub, and Spark Structured Streaming.
• Knowledge of finance and revenue data, including billing, GL, revenue recognition, reconciliation, and period-end close.
• Experience with GitHub Actions, Azure DevOps, Great Expectations, and enterprise data modeling.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.