Data Engineer

BrickRed Systems; Frisco (TX); On-site; USD 120,000 - 170,000; Full time

Job summary

BrickRed Systems in Frisco, TX is seeking an experienced Data Engineer to design and operate scalable, governed data pipelines in an Azure-native environment. You will work with PySpark, SQL, Azure Data Factory (ADF), Databricks, Python, and Snowflake to enable a third-party data enrichment platform.

The role focuses on building reliable, privacy-compliant pipelines, optimizing costs, and ensuring data quality with strong governance and access controls.

Qualifications

• 5+ years of Data Engineering experience.
• Hands-on with PySpark, SQL, and ADF.
• Experience with Databricks, ETL, Python, and Azure.
• Strong experience building and troubleshooting large-scale Spark pipelines.
• Experience with Snowflake for analytics and data warehousing.

Responsibilities

• Build and enhance ingestion pipelines for large-scale batch and event-driven data processing.
• Integrate data from third-party enrichment vendors and various platforms.
• Develop scalable ETL/ELT pipelines using PySpark, Databricks, Python, SQL, and ADF.
• Implement data validation, idempotency, replay/backfill, and deduplication strategies.
• Own monitoring, alerting, dashboarding, and operational readiness.
• Troubleshoot Spark failures and perform root cause analysis.

Skills

PySpark

SQL

ETL

Python

Snowflake

Spark

Databricks

Azure

Tools

Azure Data Factory (ADF)

Databricks

Snowflake

Job description

We are seeking an experienced Data Engineer with strong expertise in PySpark, SQL, Azure Data Factory (ADF), Databricks, ETL, Python, Azure, and Snowflake. The role focuses on building reliable, governed, scalable, privacy-compliant, and cost-efficient data pipelines for an Azure-native third-party data enrichment platform.

Required Skills

• PySpark
• SQL
• Databricks
• ETL
• Python
• Snowflake
• Spark

Experience

• 5+ years of Data Engineering experience.
• 5+ years of hands-on experience with PySpark, SQL, and ADF.
• 5+ years of experience with Databricks, ETL, Python, and Azure.
• Strong experience developing and troubleshooting large-scale Spark data pipelines.
• Experience with Snowflake for analytics and data warehousing.
• Strong understanding of data governance, privacy, compliance, and access controls.

Key Responsibilities

• Build and enhance ingestion pipelines for large-scale batch and event-driven data processing.
• Integrate data from third-party enrichment vendors, digital platforms, Conversion API (CAPI) integrations, and rewards/promotions systems.
• Develop scalable ETL/ELT pipelines using PySpark, Databricks, Python, SQL, and ADF.
• Implement data validation, idempotency, replay/backfill, and deduplication strategies.
• Own monitoring, alerting, dashboarding, and operational readiness.
• Troubleshoot Spark failures using logs and perform root cause analysis.
• Diagnose Spark performance issues involving shuffle, skew, partitioning, and workload optimization.
• Ensure pipeline reliability and SLA adherence.
• Apply privacy, compliance, and governance requirements across pipelines and datasets.
• Support Unity Catalog, lineage, access controls, and PII/non-PII handling.
• Maintain documentation for tables, schemas, catalogs, and cluster usage.
• Optimize cluster sizing, compute/storage usage, and overall pipeline cost.
• Balance cost, quality, performance, and SLA requirements.

Good to Have

Desired Competencies

• Strong analytical and troubleshooting skills.
• Strong Spark performance optimization knowledge.
• Ability to troubleshoot production data pipeline issues independently.
• Strong understanding of data quality and reliability patterns.
• Knowledge of data governance, privacy, compliance, and lineage.
• Excellent communication and collaboration skills.
• Ability to work independently with minimal supervision.
• Strong ownership mindset.
• Ability to balance cost, quality, reliability, and SLA requirements.
• Experience working in Agile environments.
• Databricks Development
• Snowflake Data Engineering
• Data Quality & Validation
• Data Governance & Compliance
• Data Lineage & Access Control
• Spark Performance Optimization
• Cost Optimization
• Technical Documentation

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.