Senior Azure Databricks Data Engineer

MasterApp Labs LLC; Indianapolis (IN); Hybrid; USD 96,000 - 165,000; Part time

Benefits offered by this job

401(k) matching

Dental insurance

Health insurance

Vision insurance

Job summary

MasterApp Labs LLC is seeking a Senior Azure Databricks Data Engineer in Indianapolis, IN. The contract role requires building and maintaining batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake within a medallion lakehouse architecture.

You will design dimensional models in Azure Synapse, orchestrate workflows with ADF and Databricks; ensure data governance and HIPAA compliance while collaborating across data teams.

Qualifications

• Master's degree or equivalent in data-related field.
• Bachelor's + 2+ years or 4+ years experience in data engineering.
• Certifications may substitute for formal education/experience.
• Databricks notebooks in PySpark, Spark SQL, and R.
• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, VACUUM.
• Medallion architecture with incremental ingestion (Auto Loader, Structured Streaming, or Lakeflow/DLT).
• Unity Catalog for permissions, lineage, storage credentials.
• Cluster sizing, SQL warehouse tuning; diagnose partitioning/shuffle/caching via Spark UI.
• ADF Pipelines, Databricks Workflows; Git repos for source control.
• MPP tables, distribution, partitioning; star/schema design.
• Parquet/Delta/CSV in ADLS Gen2 via OPENROWSET/external tables.
• STAR schema facts/dimensions; SCD Type 1/2; stored procedures.
• COPY INTO, PolyBase, CETAS; Synapse/ADF pipelines.
• Monitoring with Azure Monitor; troubleshoot failed jobs.
• Strong written and verbal communication.
• Microsoft lakehouse patterns; Synapse-Databricks interoperability.
• Databricks Certification; DP-203 as desired.
• Power BI connectivity to Synapse (Import/DirectQuery).

Responsibilities

• Assist with data strategy and data integration architecture.
• Define and implement data quality, governance, and privacy for PHI/PII.
• Design, build, and maintain batch and streaming data pipelines in Azure Databricks.
• Tune SQL workloads in Azure Synapse Analytics to support reports.
• orchestrate pipelines with Databricks Workflows and Synapse Data Factory.
• Collaborate with data architects, scientists, and analysts on designs.
• Document solutions with diagrams, flowcharts and comments.
• Support operations with system analysis and policy recommendations.
• Work with secure systems to ensure HIPAA compliance.
• Prepare and present technical details to non-technical stakeholders.

Skills

Databricks PySpark

Medallion architecture

Unity Catalog

Spark UI

ADF Pipelines

Git Repos

Star schema

COPY INTO

PolyBase

Synapse Databricks

Delta Lake

OPENROWSET

SCD Type 1/2

DLT

Azure Monitor

Power BI

Education

Master's degree in data science/analytics/CS/engineering

Bachelor's degree + 2+ years OR 4+ years hands-on data engineering

Certifications/substitute for edu/experience

Tools

Azure Databricks

Azure Synapse Analytics

Azure Data Factory

GitHub

PolyBase

Job description

Benefits

• 401(k) matching
• Dental insurance
• Health insurance
• Vision insurance

Job Title Senior Azure Databricks Data Engineer

Location Indianapolis, IN (Hybrid)-Onsite one day per week

Position Type Contract

Interview Mode Web Cam

Job Duties

• Assists with the data technology strategy, including resolvingstrategic data integration issues, such as data quality and stewardship,real-time,
• Event-based data integration, and crafting a universal vision for data integration
• Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems
• Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering
• Provides advice, training, and technical support for various projects
• Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel
• Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.
• Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture
• Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs
• Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
• Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy
• Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.
• Assists in operations by conducting systems analysis; recommendingchanges in policies and procedures.
• The job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Other duties, responsibilities and activities may change or be assigned at any time.

Required/Desired Skills

• Skill | Required /Desired | Amount | Actual Years of Experience
• Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR.... | Required | 4 |
• Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role; OR.. | Required | 4 |
• Relevant certifications and coursework may substitute for formal education/experience requirements | Required | 4 |
• Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions | Required | 4 |
• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM) | Required | 4 |
• Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT | Required | 4 |
• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials | Required | 4 |
• Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI | Required | 4 |
• Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control | Required | 4 |
• Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning | Required | 4 |
• Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views | Required | 4 |
• Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2 | Required | 4 |
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers) | Required | 4 |
• Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing | Required | 4 |
• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault | Required | 4 |
• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles | Required | 4 |
• Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs | Required | 4 |
• Strong written and verbal communication skills | Required |
• Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks | Required | 4 |
• Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus) | Highly desired |
• Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held | Highly desired |
• Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance | Desired | 4 |

Flexible work from home options available.