Data Engineer (Databricks)

Distillery; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Distillery is seeking a Data Engineer to embed within a client’s data team and migrate business logic from Power BI and Excel into a governed Databricks Lakehouse. You will reverse‑engineer existing reports, remodel logic for performance, validate results, and document definitions for transfer of knowledge to the client’s internal team.

You will design and maintain clean silver/gold layers, apply medallion architecture, and participate in data governance using Unity Catalog while enabling

Qualifications

• Strong data modeling expertise for analytics at scale.
• Ability to reverse‑engineer Power BI and Excel workflows.
• Experience implementing data governance and handling sensitive data.

Responsibilities

• Partner with business SMEs to understand reporting processes and metrics.
• Reverse‑engineer logic in Power BI and Excel and trace to source layers.
• Design and build clean silver/gold layers in Databricks with medallion architecture.
• Materialize logic in the data layer to enable fast, pre‑validated datasets.
• Support GDPR/CCPA data handling and Unity Catalog governance.
• Develop and maintain a data dictionary and semantic layer.

Skills

Databricks

PySpark

SQL

Data modeling

Unity Catalog

Delta Lake

Tools

Power BI

Excel

Job description

Distillery is a software development partner that helps organizations design, build, and scale modern technology solutions. Leading companies work with us to accelerate digital initiatives, solve complex technical challenges, and bring innovative products to market faster. Working as an extension of our clients' teams, we deliver lasting business value. Distillery is committed to diversity and inclusion. We embrace a dynamic and inclusive culture where you have the opportunity to continuously learn and grow.

About the Position

We are seeking a Data Engineer to embed within our client's data team and accelerate the migration of business logic from Power BI and Excel into a governed Databricks Lakehouse. In this role, you will work directly with business users to understand and reverse-engineer their existing reports, then re-model and rebuild that logic in Databricks with performance and scalability in mind. You will also validate results, document definitions, and progressively transfer knowledge to the client's internal team.

Responsibilities

• Partner with business SMEs (data team, finance, and other functions) to understand existing reporting processes and the metrics that must be exactly correct.
• Reverse-engineer logic embedded in Power BI (DAX, semantic model relationships) and Excel-based workflows, tracing it back to silver and source layers.
• Design and build clean, well-modeled silver and gold/reporting layers in Databricks, applying medallion architecture and coding best practices.
• Pre-calculate and materialize logic in the data layer so the visualization layer serves small, fast, pre-validated datasets.
• Contribute to data governance using Unity Catalog (access control, PII protection) and support GDPR/CCPA-aligned data handling.
• Build and maintain a data dictionary and semantic layer to support self-service and Genie/AI-assisted querying.
• Develop and optimize incremental pipelines, adding validation and reconciliation consistent with the team's existing practices.
• Support the onboarding of new data sources (e.g., Magento, CRM) into the platform.
• Pair with and upskill the client's internal engineers and analysts, documenting work for ongoing internal support.

Requirements

• Strong data modeling expertise, with the ability to judge whether an existing model reflects the single source of truth and to design the right model for performance and scale.
• Hands‑on Databricks experience: PySpark and SQL, notebooks, Delta Lake, medallion architecture, incremental processing, and Unity Catalog.
• Ability to reverse-engineer Power BI, including reading DAX and semantic model relationships; working knowledge of Power Query.
• Solid SQL skills and experience with the Azure data ecosystem (ADLS Gen2, Azure Synapse, Azure SQL/SQL Server).
• Demonstrated business fluency: comfortable working with non‑technical business users, eliciting requirements, and translating business needs into technical implementation.
• Experience implementing data governance and handling sensitive/PII data.
• Strong communication skills and a self‑directed, ownership mindset suited to an embedded staff‑augmentation role.

Nice to Have

• Experience building semantic layers and enabling natural language/LLM‑based querying (e.g., Databricks Genie).
• Familiarity with AI‑assisted development workflows (e.g., Claude, ChatGPT) for interpreting and rebuilding legacy logic.
• Exposure to Dynamics 365 F&O data structures and/or Synapse Link.
• Experience integrating e‑commerce (Magento) and CRM data sources.
• Familiarity with GDPR/CCPA data handling requirements.
• Experience with retail, e‑commerce, or consumer goods data.