Senior Azure Databricks Data Engineer (Hybrid/Remote)

MasterApp Labs LLC; Indianapolis (IN); Hybrid; USD 96,000 - 165,000; Part time

Benefits offered by this job

401(k) matching

Dental insurance

Health insurance

Vision insurance

Job summary

MasterApp Labs LLC is seeking a Senior Azure Databricks Data Engineer in Indianapolis, IN. The contract role requires building and maintaining batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake within a medallion lakehouse architecture.

You will design dimensional models in Azure Synapse, orchestrate workflows with ADF and Databricks; ensure data governance and HIPAA compliance while collaborating across data teams.

Qualifications

• Master's degree or equivalent in data-related field.
• Bachelor's + 2+ years or 4+ years experience in data engineering.
• Certifications may substitute for formal education/experience.
• Databricks notebooks in PySpark, Spark SQL, and R.
• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, VACUUM.
• Medallion architecture with incremental ingestion (Auto Loader, Structured Streaming, or Lakeflow/DLT).
• Unity Catalog for permissions, lineage, storage credentials.
• Cluster sizing, SQL warehouse tuning; diagnose partitioning/shuffle/caching via Spark UI.
• ADF Pipelines, Databricks Workflows; Git repos for source control.
• MPP tables, distribution, partitioning; star/schema design.
• Parquet/Delta/CSV in ADLS Gen2 via OPENROWSET/external tables.
• STAR schema facts/dimensions; SCD Type 1/2; stored procedures.
• COPY INTO, PolyBase, CETAS; Synapse/ADF pipelines.
• Monitoring with Azure Monitor; troubleshoot failed jobs.
• Strong written and verbal communication.
• Microsoft lakehouse patterns; Synapse-Databricks interoperability.
• Databricks Certification; DP-203 as desired.
• Power BI connectivity to Synapse (Import/DirectQuery).

Responsibilities

• Assist with data strategy and data integration architecture.
• Define and implement data quality, governance, and privacy for PHI/PII.
• Design, build, and maintain batch and streaming data pipelines in Azure Databricks.
• Tune SQL workloads in Azure Synapse Analytics to support reports.
• orchestrate pipelines with Databricks Workflows and Synapse Data Factory.
• Collaborate with data architects, scientists, and analysts on designs.
• Document solutions with diagrams, flowcharts and comments.
• Support operations with system analysis and policy recommendations.
• Work with secure systems to ensure HIPAA compliance.
• Prepare and present technical details to non-technical stakeholders.

Skills

Databricks PySpark

Medallion architecture

Unity Catalog

Spark UI

ADF Pipelines

Git Repos

Star schema

COPY INTO

PolyBase

Synapse Databricks

Delta Lake

OPENROWSET

SCD Type 1/2

DLT

Azure Monitor

Power BI

Education

Master's degree in data science/analytics/CS/engineering

Bachelor's degree + 2+ years OR 4+ years hands-on data engineering

Certifications/substitute for edu/experience

Tools

Azure Databricks

Azure Synapse Analytics

Azure Data Factory

GitHub

PolyBase

Job description

MasterApp Labs LLC is seeking a Senior Azure Databricks Data Engineer in Indianapolis, IN. The contract role requires building and maintaining batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake within a medallion lakehouse architecture.

You will design dimensional models in Azure Synapse, orchestrate workflows with ADF and Databricks; ensure data governance and HIPAA compliance while collaborating across data teams.