IN - IDOH Data Scientist

RICEFW Technologies Inc; Indianapolis (IN); Presencial; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

RiceFW Technologies Inc. seeks a data engineering professional to design and maintain secure, scalable data pipelines in Azure Databricks and Synapse. You will model data, optimize workloads, and ensure governance across PHI/PII data in line with HIPAA policies.

The role collaborates with data architects, scientists, analysts, and project managers to translate business needs into technical designs, while supporting batch and streaming integrations and documenting solutions for operational teams.

Formación

• Experience with Databricks notebooks in PySpark and Spark SQL.
• Knowledge of Delta Lake, ACID, and data lakehouse patterns.
• Ability to design and maintain batch/streaming pipelines.

Responsabilidades

• Design and implement data integration pipelines across Azure Databricks and Synapse.
• Build scalable data models and SQL workloads for reporting needs.
• Enforce data governance, security, and PHI/PII protection per HIPAA.
• Collaborate with architects, analysts, and project managers to deliver data products.

Conocimientos

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Azure Data Factory

Databricks Workflows

Data modeling

Data governance

Strong communication

Educación

Master's degree in data science or related field

Bachelor's degree + 2+ years of experience

Relevant certifications may substitute

Herramientas

Azure Synapse

Azure Databricks

Auto Loader

Structured Streaming

Descripción del empleo

Designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics.

Local candidates only will be considered for this position.

This role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional teams to deliver high quality, reliable data products for the agency.

Plays a key role in implementing the agency s data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing data governance standards. The position assists with requirements gathering, solution reviews, and technical support, and works closely with data architects, scientists, analysts, and project managers to translate business needs into technical designs. This role supports operational processes, contributes to policy and procedure improvements, and may provide mentorship or supervision based on agency needs.

Job Duties

Assists with the data technology strategy, including resolving strategic data integration issues, such as data quality and stewardship, real-time,

Event-based data integration, and crafting a universal vision for data integration

Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems

Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering

Provides advice, training, and technical support for various projects

Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel

Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation

Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.

Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture

Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs

Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery

Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy

Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.

Assists in operations by conducting systems analysis; recommending changes in policies and procedures.

The job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Other duties, responsibilities and activities may change or be assigned at any time.

Required/Desired Skills

• SkillRequired /DesiredAmountof ExperienceMaster's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR....
• Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role; OR..
• Relevant certifications and coursework may substitute for formal education/experience requirements
• Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions
• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)
• Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT
• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
• Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI
• Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control
• Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning
• Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views
• Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)
• Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing
• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
• Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs
• Strong written and verbal communication skills
• Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks
• Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus)Highly desired
• Microsoft Certified: Azure Data Engineer Associate (DP-203) if already heldHighly desired
• Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performanceDesired4Years

Questions

Question1Absences greater than two weeks MUST be approved by CAI management in advance, and contact information must be provided to CAI so that the resource can be reached during his or her absence. The Client has the right to dismiss the resource if he or she does not return to work by the agreed upon date. Do you accept this requirement?

Question2Please provide the candidate's email address.

Question3Candidate must be located in Indiana to be considered for this position. Please provide the city and state where candidate currently resides.

Question4Candidate must be paid NA for engagement. The provider markup for this position is %. Do you accept this requirement?

Question5Resource will be required to work a hybrid schedule being onsite one day per week. Please confirm the resource understands the hybrid schedule.

Question6Candidate will be required to work hours per week. Please confirm the resource understand the work hours.

Question7Candidate must meet one of the following qualifications: Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; - a Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role. Relevant certifications and coursework may substitute for formal education/experience requirements.