Data Engineer: Scalable Pipelines in Azure Databricks

BrickRed Systems; Frisco (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

BrickRed Systems is seeking a highly skilled Data Engineer to build an Azure-native third-party data enrichment platform using Databricks, Spark, Snowflake, SQL, Python, and ADF. You will develop reliable, scalable data pipelines with strong data quality, privacy, and cost efficiency.

The ideal candidate will have hands-on PySpark, SQL, ETL, Databricks, Spark troubleshooting, Snowflake, and ADF experience, supporting analytics and downstream CDP systems. In-person client interviews are mandatory.

Qualifications

• 4+ years of experience in Data Engineering or related technical roles.
• FAANG or Top Product Company background is mandatory.
• Strong hands-on experience with Data Engineering, SQL, Spark, ETL, and Python.
• 4+ years of experience with Databricks and Azure Data Factory (ADF).
• Strong proficiency in PySpark and SQL, including hands-on coding.
• Strong understanding of Apache Spark fundamentals, including partitioning and shuffle optimization.
• Experience with Databricks notebooks/jobs, performance tuning, and medallion architecture patterns.
• Experience with Snowflake for data modeling and warehousing.
• Experience with Azure Data Factory for data orchestration and Azure-native integrations.
• Knowledge of data governance, lineage, access control, PII handling, and privacy/compliance requirements.
• Ability to work independently with strong ownership and problem-solving skills.
• In-person client interview is mandatory.

Responsibilities

• Build and enhance data ingestion pipelines for large-scale batch and event-driven data paths.
• Integrate data from third-party enrichment vendors and digital platforms through CAPI and Rewards/Promotions systems.
• Develop reliable ETL pipelines using PySpark, SQL, Databricks, Snowflake, Python, and ADF.
• Implement data validation, idempotency, replay/backfill, deduplication, and auditability practices.
• Own pipeline monitoring, alerting, dashboarding, and operational readiness.
• Troubleshoot Spark failures via logs and perform root-cause analysis.
• Diagnose and optimize Spark performance issues involving shuffle, skew, and partitioning.
• Apply privacy, compliance, and data governance requirements across pipelines.
• Support governance capabilities including Unity Catalog, data lineage, and access controls.
• Maintain documentation for tables, schemas, catalogs, and cluster usage.
• Design pipelines with cost efficiency and workload optimization in mind.
• Balance cost, quality, performance, and SLA requirements in engineering decisions.
• Raise and manage data quality escalations and contribute to platform architecture.
• Collaborate within a small data engineering team and take ownership of deliverables.

Skills

Databricks

PySpark

SQL

Spark

Snowflake

Azure Data Factory

ETL

Python

Data governance

Tools

Delta Lake

Job description

BrickRed Systems is seeking a highly skilled Data Engineer to build an Azure-native third-party data enrichment platform using Databricks, Spark, Snowflake, SQL, Python, and ADF. You will develop reliable, scalable data pipelines with strong data quality, privacy, and cost efficiency.

The ideal candidate will have hands-on PySpark, SQL, ETL, Databricks, Spark troubleshooting, Snowflake, and ADF experience, supporting analytics and downstream CDP systems. In-person client interviews are mandatory.