Lead Architect

Fractal Analytics Inc; New York (NY); On-site; USD 150,000 - 200,000; Full time

Benefits offered by this job

Health, dental, vision, life insurance

Disability plans

401(k) Plan

11 paid holidays

Parental Leave (12 weeks)

PTO policy

Job summary

Fractal Analytics Inc in New York invites a Lead Architect – Azure & AI to design and deliver cloud-native data solutions on Azure and Snowflake, enabling analytics, ML/AI workloads, and governance.

You will lead end-to-end data pipelines, ETL/ELT, PySpark, SQL, Snowflake, and integrate NoSQL, while partnering with stakeholders. The role includes DevOps practices and a strong focus on data security.

Qualifications

• 8+ years in data engineering with Azure and Snowflake.
• Strong Insurance domain experience preferred.
• Proficient in SQL and Python/PySpark for data transformations.
• Experience designing ETL/ELT pipelines and data warehouses.
• Expertise with Snowflake schema design, performance tuning, and security.
• Hands-on Spark tuning and distributed processing knowledge.

Responsibilities

• Design and deliver large-scale cloud data pipelines on Azure and Snowflake.
• Develop and maintain ETL/ELT workflows for structured and unstructured data.
• Write production-grade Python (PySpark) and SQL for data transformations.
• Architect Snowflake solutions with schema design and performance tuning.
• Leverage Azure Synapse and Snowflake for analytics and data warehousing.
• Partner with stakeholders to design DataMarts and Data Lakes.
• Implement data quality, governance, and lineage across environments.
• Optimize Spark workloads on Databricks and manage Notebooks/Jobs
• Integrate NoSQL stores and hybrid Snowflake-Azure solutions
• Utilize Azure Cognitive Search for indexing and search

Skills

Azure Data Factory

Azure Synapse

Databricks

Snowflake

SQL

Python

PySpark

ETL/ELT pipelines

NoSQL (Cosmos DB/MongoDB)

Kafka

Event Hub

SFTP/API integration

Spark performance tuning

CI/CD DevOps

Tools

Azure Data Factory

Azure Synapse

Databricks

Snowflake

Elasticsearch

Azure DevOps

Jenkins

Git

Cosmos DB

MongoDB

Kafka

Event Hub

SFTP

APIs

CI/CD pipelines

Job description

Role Overview

We are looking for a Lead Architect – Azure & AI to join our growing data engineering practice. In this role, you will architect and deliver large-scale, cloud-native data solutions on Azure and Snowflake, enabling advanced analytics, ML/AI workloads, and enterprise data governance.

Key Responsibilities

• Design, build, and optimize end-to-end data pipelines leveraging Azure Data Factory, Databricks, and Snowflake for large-scale data ingestion, transformation, and loading.
• Develop and maintain ETL/ELT workflows for structured, semi-structured (JSON, XML, Parquet), and unstructured data from on-premises systems, cloud platforms, APIs, flat files, RDBMS, and streaming sources.
• Write efficient, production-grade code in Python (PySpark) and SQL for advanced data transformations, workflow orchestration, and automation.
• Architect scalable Snowflake solutions including schema design, Virtual Warehouse configuration, query performance tuning, and cost-efficient data storage and retrieval strategies.
• Leverage Azure Synapse and Snowflake for advanced analytics and data warehouse solutions, including MPP (Massively Parallel Processing) architectures.
• Partner with business and technology stakeholders to design DataMart and Data Lake solutions supporting analytics, reporting, and ML/AI use cases.
• Implement data quality frameworks, governance policies, lineage tracking, and metadata management across multiple environments.
• Optimize Spark workloads on Databricks for distributed processing at scale; deploy and manage reusable Notebooks and Jobs in production.
• Integrate NoSQL stores (Cosmos DB, MongoDB, etc.) and build hybrid solutions spanning Snowflake and Azure services.
• Leverage Azure Cognitive Search and Elasticsearch for indexing and unstructured data search integration.
• Drive DevOps best practices including CI/CD pipelines, source control (Git), and automated deployments via Azure DevOps and Jenkins.
• Ensure adherence to enterprise data security, cataloging, and access-control policies across all data assets.

Required Skills

• 8+ years of experience as a Data Engineer with deep expertise in Azure Data Services (ADF, Synapse, Databricks) and Snowflake.
• Strong background in the Insurance domain with hands‑on Azure Data Engineering experience (ADB + PySpark/Python + Snowflake).
• Proficiency in SQL (complex queries, stored procedures, functions, views) and Python/PySpark programming.
• Proven experience designing and implementing ETL/ELT pipelines with Azure Data Factory and orchestrating complex, multi‑step workflows.
• Comprehensive Snowflake expertise: Star/Snowflake schema design, role‑based security, performance tuning, query optimization, and pipeline features (Snowpipe, Tasks, Streams).
• Solid understanding of Spark architecture with hands‑on experience in performance tuning, data partitioning, and distributed processing.
• Experience integrating data from RDBMS, SFTP, APIs, Kafka, Event Hub, and big‑data platforms (Hadoop, Hive).
• Familiarity with NoSQL database design (Cosmos DB or equivalent) and streaming data pipeline patterns.
• Knowledge of ML/AI data preparation workflows; ability to refactor traditional ML code for distributed execution on Spark/Databricks.
• Strong command of Agile methodologies, DevOps practices, and CI/CD deployment frameworks.

Preferred Skills

• Exposure to data search and indexing technologies such as Elasticsearch or Azure Cognitive Search.
• Azure Data Engineer Associate certification (DP-200/201/203) or SnowPro Core/Advanced Architect certification.
• Experience working on data platforms serving analytics and AI use cases in large enterprise environments.

Pay

The wage range for this role is $150,000 – $200,000, adjusted for geographic differential and other factors. You may also be eligible for a discretionary bonus for the current performance period.

Benefits

As a full‑time employee or a salaried employee working more than 30 hours per week, you will be eligible to participate in health, dental, vision, life insurance, and disability plans in accordance with the plan documents. You will be eligible for benefits on the first day of employment with the Company. Additionally, you are eligible to participate in the Company 401(k) Plan after 30 days of employment. The Company provides 11 paid holidays, 12 weeks of Parental Leave, and a “free time” PTO policy that allows flexibility for sick time or vacation.

EEO Statement

Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.