Azure Data Engineer

Skywalk Global; Indianapolis (IN); On-site; USD 110,000 - 150,000; Full time

Job summary

Skywalk Global is seeking an Azure Data Engineer for an onsite role in Indianapolis to support reliable data products across integration, pipelines, modeling, governance, and documentation.

The role involves designing batch and streaming pipelines with Azure Databricks (PySpark, Spark SQL, Delta Lake), maintaining dimensional models in Azure Synapse, and ensuring HIPAA-compliant data governance. Collaboration with data architects, scientists, and analysts is essential.

Qualifications

• 4+ years of experience in data engineering.
• Master’s degree or Bachelor’s degree with 2+ years of relevant experience.
• Databricks notebooks using PySpark, Spark SQL, and R.
• Experience with CID transactions, MERGE upserts, schema enforcement/evolution, time travel, and OPTIMIZE.
• Medallion architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakehouse patterns.
• Working with Unity Catalog for permissions, lineage, and security.

Responsibilities

• Support data technology strategy and data quality stewardship.
• Oversee design and implementation of data solutions for integration and synthesis.
• Define and maintain data management policies and engineering standards.
• Provide training and technical support for projects.
• Gather requirements and explain technical concepts to stakeholders.
• Collaborate with data architects, scientists, and analysts on solution design.
• Extract, transform, and load data from diverse sources.
• Design and maintain batch and streaming Azure Databricks pipelines using PySpark, Spark SQL, and Delta Lake.
• Develop and tune dimensional data models and SQL workloads in Azure Synapse Analytics.
• Orchestrate workflows with Databricks Workflows/Jobs and Synapse or ADF pipelines; implement scheduling, alerting, and recovery.
• Apply HIPAA-compliant data governance and Unity Catalog security controls.
• Document solutions with diagrams, flowcharts, and well-commented code.
• Support operations by analyzing systems and recommending policy changes.

Skills

Data engineering

Communication

Problem solving

Education

Master’s degree in data science, analytics, informatics, CS, mathematics, engineering, or related

Bachelor’s degree + 2+ years experience

4+ years hands-on experience in data engineering

Tools

Azure Databricks

PySpark

Spark SQL

Delta Lake

Azure Synapse Analytics

Databricks Workflows/Jobs

Azure Data Factory

OPENROWSET

Parquet/Delta/CSV

ADLS Gen2

Hive/Hive-like data lake patterns

Unity Catalog

Job description

Skywalk Global is seeking an Azure Data Engineer for an onsite role supporting reliable data products across integration, pipelines, modeling, governance, and documentation.

Responsibilities

• Support data technology strategy, including resolving data integration issues related to data quality and stewardship across secure systems
• Provide direction and oversight for design and implementation of data solutions for integration and synthesis across multiple secure systems
• Assist in defining, implementing, and maintaining data management policies and engineering design standards
• Advise, train, and provide technical support for projects
• Participate in requirements gathering, solution reviews, and explaining technical complexities to non-technical stakeholders
• Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Extract, transform, and load data from a variety of sources using data engineering tools and methods
• Design, build, and maintain batch and streaming Azure Databricks pipelines using PySpark, Spark SQL, and Delta Lake following medallion (bronze/silver/gold) architecture
• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboards
• Orchestrate and monitor workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
• Apply data governance and security controls to protect PHI/PII in line with HIPAA and agency policy (including Unity Catalog permissions, role-based access, row- and column-level security, and data masking)
• Document and demonstrate solutions through documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code
• Support operations by performing systems analysis and recommending changes in policies and procedures

Requirements

• 4+ years experience required, supported by one of the following education/experience paths: a Master’s degree in a related field (data science, analytics, informatics, computer science, mathematics, engineering, or related) OR a Bachelor’s degree plus 2+ years relevant experience OR 4+ years hands‑on experience in a similar data engineering role
• Relevant certifications and coursework may substitute for formal education/experience requirements (still requiring 4 years)
• Developing with Databricks notebooks using PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions
• Experience with CID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance such as OPTIMIZE, Z‑order, and VACUUM
• Working with medallion architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT
• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
• Right‑sizing clusters and SQL warehouses and diagnosing partitioning, shuffle, skew, and caching issues using Spark UI
• Scheduling and managing dependencies with ADF Pipelines and Databricks Workflows (Jobs), including using Git folders (Repos) for source control
• Designing MPP warehouse tables, including distribution (hash, round‑robin, replicated), columnstore indexes, and partitioning
• Querying Parquet, Delta, and CSV in ADLS Gen2 using OPENROWSET, external tables, and views
• Building star‑schema facts and dimensions, plus stored procedures and views, including SCD Type 1 and SCD Type 2
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)
• Experience with workload management, resource classes, statistics, materialized views, result‑set caching, and pool scaling/pausing
• ADLS Gen2 folder/file‑format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
• Monitoring pipelines and workloads with Azure Monitor/Log Analytics, alerting, and troubleshooting failed jobs
• Strong written and verbal communication skills
• Understanding Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

Location

• Indianapolis, IN (onsite)
• Local candidates only will be considered

Technologies

• Azure Databricks, PySpark, Spark SQL, Delta Lake
• Medallion (bronze/silver/gold) lake house architecture
• Azure Synapse Analytics, SQL pools
• Databricks Workflows/Jobs, Azure Data Factory
• Unity Catalog, HIPAA, Unity Catalog permissions, role‑based access, row- and column‑level security, data masking
• Databricks notebooks, R
• CID transactions, MERGE upserts, schema enforcement/evolution, time travel, OPTIMIZE, Z‑order, VACUUM
• Auto Loader, Structured Streaming, Lakeflow/DLT
• Spark UI, ADF Pipelines
• MPP warehouse tables, Parquet, Delta, CSV, ADLS Gen2, OPENROWSET, external tables, views
• Star‑schema, SCD Type 1, SCD Type 2, stored procedures
• COPY INTO, PolyBase, CETAS, data flows, triggers
• Materialized views, result‑set caching, Azure Monitor, Log Analytics
• Entra ID, managed identities, Key Vault
• Azure DevOps, GitHub, Git (Repos), Databricks Asset Bundles
• Microsoft lakehouse patterns, Power BI