Data Scientist

Technotopia Solutions LLC; Indianapolis (IN); Híbrido; USD 49.000 - 70.000; Jornada completa

Descripción de la vacante

State of Indiana seeks an IN - IDOH Data Scientist to design and implement scalable data pipelines in Azure Databricks and Synapse, delivering reliable data products while enforcing governance for PHI/PII data.

You will collaborate with data architects, scientists, analysts and project managers to translate business needs into robust technical designs, supporting both batch and streaming workloads in a regulated environment.

Formación

• Master's degree or 4+ years in relevant field with data engineering experience.
• 4+ years developing Databricks notebooks with PySpark, Spark SQL, and R.
• 4+ years with Delta Lake features and medallion architecture (Bronze/Silver/Gold).
• Strong written and verbal communication skills.

Responsabilidades

• Design, build, and maintain batch and streaming data pipelines in Azure Databricks.
• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse.
• Govern data security controls and HIPAA compliance for health data.
• Collaborate with data architects, scientists, analysts and PMs on solution design.

Conocimientos

PySpark

Spark SQL

Delta Lake

Azure Databricks

Azure Synapse

Data governance

SQL workloads

ETL pipelines

Educación

Master's degree in Data Science or related field

Herramientas

Databricks

Azure Data Factory

Synapse Analytics

Delta Lake

Descripción del empleo

Job: IN - IDOH Data Scientist (812974)

Client: State of Indiana

Duration: 1 Year Contract

Location: 2 N Meridian St, Indianapolis, IN 46204 - Hybrid

Interview: Webcam only

Rate: $43.35/hr

Designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics.

Local candidates only will be considered for this position.

This role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional teams to deliver high quality, reliable data products for the agency.

Plays a key role in implementing the agency's data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing data governance standards. The position assists with requirements gathering, solution reviews, and technical support, and works closely with data architects, scientists, analysts, and project managers to translate business needs into technical designs. This role supports operational processes, contributes to policy and procedure improvements, and may provide mentorship or supervision based on agency needs.

Job Duties

• Assists with the data technology strategy, including resolvingstrategic data integration issues, such as data quality and stewardship,real-time, Event-based data integration, and crafting a universal vision for data integration
• Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems
• Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering
• Provides advice, training, and technical support for various projects
• Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel
• Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.
• Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture
• Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs
• Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
• Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy
• Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.
• Assists in operations by conducting systems analysis; recommendingchanges in policies and procedures.
• The job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Other duties, responsibilities and activities may change or be assigned at any time.

Required Skills and Qualifications

• Master's degree in Data Science, Analytics, Informatics, Computer Science, Mathematics, Engineering, or a related field; or a bachelor's degree in a related field with 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role.
• Relevant certifications and coursework may substitute for formal education or experience requirements.
• 4+ years of experience developing Databricks notebooks using PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions.
• 4+ years of experience with Delta Lake features, including ACID transactions, MERGE upserts, schema enforcement and evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM).
• 4+ years of experience with medallion architecture (Bronze/Silver/Gold) and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.
• 4+ years of experience using Unity Catalog for workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials.
• 4+ years of experience optimizing Databricks clusters and SQL warehouses, diagnosing partitioning, shuffle, data skew, and caching issues using Spark UI.
• 4+ years of experience scheduling jobs and managing dependencies using Azure Data Factory (ADF) Pipelines and Databricks Workflows (Jobs), with Git folders (Repos) for source control.
• 4+ years of experience designing MPP warehouse tables, including hash, round-robin, and replicated distributions, columnstore indexes, and partitioning.
• 4+ years of experience querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views.
• 4+ years of experience building star-schema fact and dimension tables, stored procedures, and views, including Slowly Changing Dimensions (SCD) Types 1 and 2.
• 4+ years of experience developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines using copy activities, data flows, and triggers.
• 4+ years of experience with workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing.
• 4+ years of experience with ADLS Gen2 folder and file-format design (Parquet/Delta), Entra ID access control, managed identities, and Azure Key Vault.
• 4+ years of experience with Git and CI/CD using Azure DevOps or GitHub, including branching, pull requests (PRs), and deployment of Synapse artifacts and Databricks Asset Bundles.
• 4+ years of experience monitoring pipelines and workloads using Azure Monitor and Log Analytics, configuring alerts, and troubleshooting failed jobs.
• Strong written and verbal communication skills.
• 4+ years of experience implementing Microsoft lakehouse patterns, including interoperability and migration paths between Azure Synapse Analytics and Databricks.

Highly Desired Qualifications

• Databricks Certified Data Engineer Associate certification; Databricks Certified Data Engineer Professional certification is a plus.
• Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.

Desired Skills

• 4+ years of experience with Power BI connectivity to Synapse, including Import versus DirectQuery modes and understanding how data model design affects dashboard performance.