Azure Data Engineer

Ventois Inc.; Shrewsbury (MA); Hybrid; USD 70,000 - 90,000; Full time

Job summary

Ventois Inc. in Shrewsbury, Massachusetts is seeking a data analyst to manage and analyze large healthcare datasets. The ideal candidate should have expertise in Python (PySpark) and SQL, with experience in Azure Data Factory and Databricks. Responsibilities include developing data models, ensuring data quality, and collaborating with stakeholders to drive data-driven decisions. This role requires a strong analytical mindset and the ability to communicate findings effectively. The company encourages continuous learning of new technologies and methodologies.

Qualifications

• Extensive experience in Python (PySpark) for data manipulation and analysis.
• Strong knowledge of SQL for querying and managing large datasets.
• Ability to apply data mining and statistical techniques in analytical projects.

Responsibilities

• Analyze large structured and semi-structured datasets using Python and SQL.
• Develop data models and analytical datasets for reporting and business intelligence.
• Implement data quality frameworks and ETL/ELT processes.

Skills

Python (PySpark)

SQL

Data mining

Statistical analysis

Azure Data Factory

Azure Databricks

Data governance

ETL/ELT processes

Data quality frameworks

Azure DevOps

Tools

Azure Data Lake Storage

Spark SQL

Azure monitoring tools

APIs

Job description

Responsibilities

• Analyze, manipulate, and process large sets of structured and semi-structured data including healthcare datasets (claims, patient, and provider data) using Python (PySpark), SQL, and Spark SQL within the Azure data platform.
• Apply data mining, data modeling, and statistical analysis techniques to extract and analyze information from large datasets stored in Azure Data Lake Storage (ADLS) and Delta Lake across Bronze, Silver, and Gold layers.
• Identify business problems and management objectives that can be addressed through data analysis; translate business requirements into analytical workflows and data solutions using Azure Data Factory and Azure Databricks.
• Develop and maintain data models and analytical datasets to support reporting, business intelligence, and downstream data consumption; identify relationships, trends, and factors that could affect the results of analysis.
• Apply feature selection and data profiling methods to identify patterns, anomalies, and data quality issues that may impact analytical outcomes; recommend data-driven improvements.
• Clean, manipulate, and prepare raw data for analysis using ETL/ELT processes; implement data quality frameworks including validation rules, cleansing, and reconciliation techniques to ensure accuracy and consistency.
• Write new functions and applications in Python (PySpark) and SQL to conduct data transformation, validation, and analysis of large-scale datasets; optimize code for performance and efficiency.
• Test, validate, and reformulate data pipelines and datasets to ensure accurate and reliable data delivery; monitor, troubleshoot, and resolve data and performance issues using Azure monitoring tools.
• Analyze data patterns and recommend data-driven solutions to key stakeholders; deliver oral and written presentations of data analysis findings to management and end users to support informed decision-making.
• Support data governance, security, and compliance practices aligned with HIPAA requirements; prepare documentation for data processes and analytical workflows to support governance and audits.
• Develop and manage workflow orchestration, scheduling, and automation for consistent and timely data delivery using Azure-based tools.
• Implement version control and deployment processes using Azure DevOps and Git to support reproducible and reliable data workflows.
• Integrate data from multiple sources including APIs, databases, and flat files into a unified, analytics-ready data platform.
• Read technical articles, research publications, and conference papers to identify emerging analytic trends and technologies; continuously evaluate new tools and methodologies to improve data processing, analysis, and platform efficiency.
• Collaborate with business stakeholders, analysts, and data scientists to gather requirements, advise on appropriate analytical techniques, and recommend data-driven solutions aligned with organizational objectives.