Cloud Data Platform Architect

Insight Global; Town of Florida (NY); On-site; USD 150,000 - 210,000; Full time

Job summary

Insight Global seeks a Senior Cloud Data Engineer to design and operate enterprise-scale data pipelines in an Azure and Snowflake environment. You will own data lifecycle processes, build scalable platforms, and automate pipelines with Terraform and CI/CD practices.

You will collaborate with stakeholders across data, BI, security, and cloud ops, driving stability, security, and performance while exploring AI-enabled capabilities within the platform.

Qualifications

• 10+ years of software design and architecture across database, Big Data, and cloud platforms.
• 5+ years of programming with Python, Java, and/or Scala.
• 3+ years of Big Data/distributed systems experience, including Azure/AWS and Snowflake.
• CI/CD using Azure DevOps and/or GitHub.
• Infrastructure as Code with Terraform.
• Data warehousing and Business Intelligence experience.
• Experience researching emerging technologies and building roadmaps.
• Strong communication, leadership, and stakeholder management skills.

Responsibilities

• Design, implement, test, deploy, and support enterprise-scale data pipelines in Azure and Snowflake.
• Own end-to-end data lifecycle from ingestion to user consumption.
• Build reusable platforms and frameworks for production-aligned data engineering.
• Develop automation, CI/CD patterns, and IaC with Terraform.
• Collaborate with data, BI, and cloud ops teams to ensure security, scalability, and performance.
• Evaluate and enable AI capabilities within the Cloud Data Platform and guide governance.

Skills

Python

Java

Scala

Azure

Snowflake

CI/CD

Terraform

Data Warehousing

Databricks

Kafka

SQL

Hadoop

Tools

Azure Data Factory

Databricks

SQL

GitHub

Azure DevOps

Job description

A client of Insight Global is seeking an experienced Senior Cloud Data Engineer / Data Platform Engineer to design, implement, test, deploy, and support enterprise-scale data pipelines within a modern Azure and Snowflake environment. This individual will own solutions across the full data lifecycle, from raw data acquisition and ingestion through transformation, curation, and end-user consumption. The engineer will develop scalable, repeatable, maintainable, and highly automated solutions leveraging Azure Data Factory, Azure Functions, Databricks, Snowflake, SQL, and related Big Data technologies. This person will work with data originating from multiple sources and formats, including relational databases, NoSQL databases, files, Kafka, and Big Data platforms. This role will have a strong focus on building reusable platforms and frameworks that provide consistent, verifiable, and automated application and infrastructure management between non-production and production environments. The engineer will support a Cloud DevOps model centered around automation, repeatable engineering patterns, CI/CD, Git, Azure DevOps/GitHub, and Infrastructure as Code using Terraform. The engineer will partner closely with business stakeholders, Cloud administrators, system teams, product teams, data warehouse teams, Business Intelligence teams, and advanced analytics groups to gather requirements and design solutions that improve stability, scalability, security, availability, and performance. Responsibilities also include developing monitoring capabilities, supporting enterprise Cloud security standards, troubleshooting production issues, escalating issues to internal teams and vendors, and participating in a rotational on-call schedule. A major component of this position will involve evaluating and enabling AI capabilities within the Cloud Data Platform. The engineer will research emerging Snowflake and Azure AI capabilities, including Snowflake Cortex, Cortex AI Functions, Cortex Search, Cortex Analyst, Cortex Code Assistant, and Azure AI services. This person will help determine how AI can improve data engineering productivity, Snowflake development, pipeline optimization, documentation, testing, monitoring, data quality, metadata management, and platform automation. The engineer will establish best practices and governance patterns for secure, responsible, and cost-effective AI adoption. This includes understanding AI cost drivers such as token consumption, model selection, Snowflake warehouse consumption, data volume, prompt design, and repeated processing. The role will also reinforce strong foundational data engineering practices, including data cleansing, standardization, curation, semantic layers, reusable pipeline design, and metadata-driven frameworks. This individual will serve as a technical business partner across multiple teams, lead complex design and process initiatives, and provide technical direction across projects. They will help develop project plans involving financial considerations, resource management, and risk while partnering with leadership on the strategic direction of a modern enterprise data platform. Strong presentation and communication skills are essential, as this individual will communicate complex technical concepts to both engineering teams and organizational leadership.

REQUIRED SKILLS AND EXPERIENCE:

• 10+ years of professional experience in software design and architecture across database, Big Data, and cloud platforms
• 5+ years of programming experience with Python, Java, and/or Scala
• 3+ years of Big Data/distributed systems experience, including Azure/AWS and Snowflake
• CI/CD using Azure DevOps and/or GitHub
• Infrastructure as Code experience with Terraform
• Data warehousing and Business Intelligence experience
• Experience researching emerging technologies and translating findings into technology roadmaps
• Strong communication, presentation, technical leadership, and stakeholder management skills

NICE TO HAVE SKILLS AND EXPERIENCE:

• Snowflake Cortex, Cortex AI Functions, Cortex Search, Cortex Analyst, and Cortex Code Assistant
• AI-ready data products, semantic layers, metadata frameworks, and AI governance
• Understanding of AI cost optimization, including token usage, model selection, compute/warehouse consumption, and prompt design
• Spark, NiFi, Impala, Sqoop, Hive, Hadoop/Cloudera
• Parquet, AVRO, ORC, CSV, and JSON
• Healthcare industry experience, HL7, and Epic data model knowledge
• Performance tuning within Snowflake, ADF, and Big Data environments
• Automated testing frameworks for data pipelines