Data Engineer: Azure, Databricks & Snowflake | Onsite/Hybrid

Tech Mirrors; Overland Park (KS); On-site; Full time

Job description

Role

Data Engineer (Azure | Databricks | Snowflake)

Location: Overland Park, KS (Onsite/Hybrid)

Key Responsibilities

• Build and enhance ingestion pipelines for large batch and event‑driven paths, integrating data from third‑party enrichment vendors, Digital platforms via Conversion API, Rewards/Promotions systems, and other sources.
• Ensure data quality, reliability & operations by implementing validation, idempotency, replay/backfill strategies and deduplication to prevent quality drift.
• Own monitoring, alerting, dashboarding and operational readiness, including wrappers around core pipelines.
• Troubleshoot failures with root‑cause analysis, interpreting Spark logs, diagnosing performance issues (shuffle, skew, partitioning) and improving stability and SLA adherence.
• Apply privacy, compliance and governance requirements across pipelines and datasets, supporting standards such as Unity Catalog, lineage, access controls, and PII handling.
• Design pipelines with cost awareness from day one: cluster sizing, workload tuning, efficient compute/storage usage and balancing cost vs quality vs SLA.
• Work collaboratively in a small, fast‑moving team, self‑driven and ownership‑oriented, raising and managing data quality escalations when issues are detected.

Required Skills

• Strong coding in PySpark and SQL with hands‑on experience.
• Databricks: notebooks, jobs, performance tuning fundamentals, medallion patterns; Spark fundamentals (partitioning, skew/shuffle optimization, log analysis).
• Snowflake: data modeling and usage for analytics and warehousing workloads.
• Azure ecosystem: Azure Data Factory orchestration and Azure‑native integrations.
• Data engineering reliability patterns: validation, idempotency, replay/backfills, deduplication, auditability.
• Data governance: Unity Catalog (preferred), lineage, access control patterns, PII handling.
• Ownership mindset: ability to execute independently without constant approvals or check‑ins.

Nice‑to‑Have Skills

• Event‑driven/streaming ingestion exposure, including Delta Live Tables (DLT).
• Experience building config‑driven export frameworks for downstream consumers/vendors.
• Interest in identity resolution concepts.
• Operational telemetry: dashboards, alerts, SLA monitoring.

Success Criteria

Ships reliable, well‑governed datasets with strong data quality practices; scales pipelines for very large volumes; prevents silent failures where quality degrades; balances delivery speed with compliance, governance, and cost controls.