Data Scientist

Millenniumsoft; Indianapolis (IN); On-site; USD 55,000 - 66,000; Full time

Job summary

Millenniumsoft in Indianapolis, IN is seeking a Data Scientist/Data Engineer to design, build, and maintain secure data solutions across Azure Databricks and Azure Synapse Analytics. The role focuses on data integration, governance, and scalable batch/streaming pipelines to deliver reliable data products for the agency.

The candidate will collaborate with cross-functional teams, implement a medallion lakehouse approach, and ensure compliance with HIPAA and agency policy while supporting

Qualifications

• Master's degree in data science, analytics, informatics, CS, math, engineering or related field.
• 4+ years hands-on data engineering experience or a relevant combination of education and experience.
• Certifications may substitute for formal requirements.
• Experience developing in PySpark, Spark SQL, and R.

Responsibilities

• Design, build, and maintain secure, scalable data solutions across Azure Databricks and Synapse.
• Develop batch and streaming data pipelines and optimize data warehouse workloads.
• Enforce data governance and security controls (PHI/PII) per HIPAA and policy.
• Collaborate with data architects, scientists, analysts, and PMs on solution design.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Data governance

Batch/Streaming pipelines

Azure Synapse Analytics

Databricks Workflows

SQL workloads

Education

Master's degree in data science/analytics/CS

Bachelor's degree + 2+ years data engineering

Tools

Azure Databricks

Azure Synapse Analytics

Delta Lake

Unity Catalog

ADF Pipelines

Job description

Title/Role: IT : DATSCIV1 : Data Scientist: DSBB

Location:Indianapolis, IN
Duration: 12 months

Note

Pay Rate: $43.35/hr

Job Short Description

• Designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics.

Job Complete Description

Local candidates only will be considered for this position.

• This role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross‑functional teams to deliver high‑quality, reliable data products for the agency.


• Plays a key role in implementing the agency’s data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing data governance standards.


• The position assists with requirements gathering, solution reviews, and technical support, and works closely with data architects, scientists, analysts, and project managers to translate business needs into technical designs.


• This role supports operational processes, contributes to policy and procedure improvements, and may provide mentorship or supervision based on agency needs.

Job Duties

• Assists with the data technology strategy, including resolving strategic data integration issues, such as data quality and stewardship,real-time,


• Event-based data integration, and crafting a universal vision for data integration


• Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems


• Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering


• Provides advice, training, and technical support for various projects


• Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel


• Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation


• Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.


• Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture


• Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs


• Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery


• Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy


• Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.


• Assists in operations by conducting systems analysis; recommending changes in policies and procedures.


• The job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Other duties, responsibilities and activities may change or be assigned at any time.

Skills Required

• Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR.... - 4 Years


• Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands‑on experience in a similar data engineering role; OR.. - 4 Years


• Relevant certifications and coursework may substitute for formal education/experience requirements - 4 Years


• Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions - 4 Years


• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM) - 4 Years


• Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT - 4 Years


• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials - 4 Years


• Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI - 4 Years


• Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control - 4 Years


• Designing MPP warehouse tables, including distribution (hash, round‑robin, replicated), columnstore indexes, and partitioning - 4 years


• Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views - 4 years


• Building star‑schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2 - 4 Years


• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers) - 4 years


• Workload management, resource classes, statistics, materialized views, result‑set caching, and pool scaling/pausing - 4 Years


• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault - 4 Years


• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles - 4 Years


• Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs - 4 Years


• Strong written and verbal communication skills


• Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks - 4 Years

Skills Highly Desired

• Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus)


• Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held


• Power BI connectivity to Synapse (Import vs. Direct Query) and how model design affects dashboard performance - 4 Years