Data Scientist

VIVA USA Inc; Indianapolis (IN); Hybrid; USD 110,000 - 160,000; Full time

Job summary

VIVA USA Inc. is seeking a skilled Data Engineer to design, build, and maintain secure data pipelines across Azure Databricks and Azure Synapse Analytics.

You’ll support data integration, pipeline development, governance, and documentation while collaborating with cross-functional teams to deliver reliable data products and robust data solutions. Responsibilities include batch and streaming data pipelines, dimensional modeling in Synapse, and enforcing data governance standards with PHI/PII

Qualifications

• Master's degree in data science, analytics, informatics, CS, math, engineering, or related field; OR 4+ years of hands-on data engineering.
• Bachelor's degree plus 2+ years of relevant experience; or 4+ years in a similar data engineering role.
• Databricks notebooks in PySpark, Spark SQL, and R including transformations, joins, aggregations, window functions.
• ACID, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance.
• Medallion architecture and ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT.
• Unity Catalog usage for workspace catalogs, schemas, permissions, lineage; external locations, storage credentials.
• QL optimization: partitioning, shuffle, caching, Spark UI diagnostics.
• ADLS Gen2 Parquet/Delta, data governance, security controls, HIPAA-compliant handling.

Responsibilities

• Develop batch and streaming data pipelines in Azure Databricks (PySpark, Spark SQL, Delta Lake).
• Load and tune dimensional models and SQL workloads in Azure Synapse (dedicated/serverless) for reporting.
• Orchestrate workflows with Databricks Workflows and Synapse/ADF pipelines; schedule and monitor jobs.
• Define and enforce data governance policies and security controls for PHI/PII.
• Collaborate with data architects, scientists, analysts, and project managers to translate business needs into designs.
• Document solutions with flowcharts, diagrams, and readable code comments.

Skills

Databricks PySpark

Spark SQL

Delta Lake

Azure Databricks

Unity Catalog

Medallion architecture

Auto Loader/Structured Streaming

Azure Data Factory

Azure Synapse Analytics

Power BI

Git/GitHub CI/CD

Databricks Workflows

Education

Master's degree in data science / analytics

Bachelor's degree + 2+ years data engineering experience

Databricks Certifications (optional)

Tools

Azure Synapse

Azure Data Factory

Power BI connectivity

Job description

Hybrid

Designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics. This role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional teams to deliver high-quality, reliable data products for the client. Plays a key role in implementing theclient’s data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing data governance standards. The position assists with requirements gathering, solution reviews, and technical support, and works closely with data architects, scientists, analysts, and project managers to translate business needs into technical designs. This role supports operational processes, contributes to policy and procedure improvements, and may provide mentorship or supervision based onclientneeds.

Job Duties

• Assists with the data technology strategy, including resolvingstrategic data integration issues, such as data quality and stewardship,real-time,
• Event-based data integration, and crafting a universal vision for data integration
• Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems
• Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering
• Provides advice, training, and technical support for various projects
• Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel
• Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.
• Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture
• Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs
• Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
• Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA andclientpolicy
• Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.
• Assists in operations by conducting systems analysis; recommendingchanges in policies and procedures.

Skill - Required / Desired - Amount of Experience

• Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR.... - Required - 4 Years
• Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role; OR.. - Required - 4 Years
• Relevant certifications and coursework may substitute for formal education/experience requirements - Required - 4 Years
• Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions - Required - 4 Years
• ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM) - Required - 4 Years
• Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT - Required - 4 Years
• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials - Required - 4 Years
• Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI - Required - 4 Years
• Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control - Required - 4 Years
• Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning - Required - 4 Years
• Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views - Required - 4 Years
• Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2 - Required - 4 Years
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers) - Required - 4 Years
• Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing - Required - 4 Years
• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault - Required - 4 Years
• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles - Required - 4 Years
• Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs - Required - 4 Years
• Strong written and verbal communication skills - Required -
• Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks - Required - 4 Years
• Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus) - Highly desired -
• Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held - Highly desired -
• Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance - Desired - 4 Years

Notes

Hybrid

VIVA is an equal opportunity employer. All qualified applicants have an equal opportunity for placement, and all employees have an equal opportunity to develop on the job. This means that VIVA will not discriminate against any employee or qualified applicant on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status