Azure Data Engineer: Databricks & Synapse Pipelines

Millenniumsoft

Indianapolis (IN)

On-site

USD 49,000 - 70,000

Full time

12 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Millenniumsoft in Indianapolis, IN is seeking a Data Scientist for a 12-month contract on W2. The role focuses on designing, building, and maintaining data solutions using Azure Databricks and Azure Synapse Analytics to support agency data needs.

The candidate will handle data integration, pipelines, governance, and documentation while collaborating with cross-functional teams to deliver scalable data products. Local candidates only; US citizens/GC holders required.

Qualifications

Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR.... - 4 Years
Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions - 4 Years
ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM) - 4 Years
Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT - 4 Years
Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials - 4 Years
Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI - 4 Years
Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control - 4 Years
Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning - 4 years
Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views - 4 years
Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2 4 Years
Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers) - 4 years
Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing - 4 Years
ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault - 4 Years
Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles - 4 Years
Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs - 4 Years
Strong written and verbal communication skills
Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks - 4 Years

Responsibilities

Assists with the data technology strategy, including resolving strategic data integration issues, such as data quality and stewardship,real-time,
Event-based data integration, and crafting a universal vision for data integration
Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems
Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering
Provides advice, training, and technical support for various projects
Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel
Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.
Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture
Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs
Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy
Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.
Assists in operations by conducting systems analysis; recommending changes in policies and procedures.
The job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Other duties, responsibilities and activities may change or be assigned at any time.