Senior Data Engineer

Alpha Business Solutions; Chicago (IL); Remote; USD 180,000 - 240,000; Full time

Job summary

Alpha Business Solutions seeks a Senior AI Data Engineer to join the Data Engineering team. You will design, build, and operate scalable ingestion and transformation pipelines in Snowflake using AWS services and Python.

You will implement Snowflake Cortex AI workflows and develop Snowflake Native Apps to package data and AI capabilities. You will develop robust prompt versioning, response parsing, validation, and fallback logic, while tracking KPIs and ensuring data quality.

Qualifications

• 8+ years of experience with at least 5 years in Data Engineering.
• 2+ years hands-on with Generative AI applications.
• Strong Snowflake expertise including advanced SQL, Tasks/Streams, and performance tuning.
• Experience with Snowflake Cortex AI or equivalent.
• Experience building Snowflake Native Apps using Snowpark with Python/Scala.
• Strong AWS experience (S3, Lambda, Glue, Step Functions, DynamoDB, EC2, CloudFormation, Kinesis).
• CI/CD tools (GitHub Actions, Jenkins) and IaC (Terraform / CloudFormation).
• Experience building structured extraction pipelines from free-text inputs.
• Experience with ML/LLM observability (Arize or similar).
• Strong evaluation design, drift monitoring, and error analysis.
• Ability to work independently and communicate with stakeholders.

Responsibilities

• Design and implement ingestion pipelines from multiple sources into Snowflake using AWS and Python.
• Build and optimize SQL-based transformations for text cleansing, normalization, and deduplication.
• Implement Snowflake Cortex AI workflows for classification, extraction, and data generation.
• Develop prompt versioning, response parsing, validation, and fallback logic.
• Snowflake Native Apps development: packaging data/AI capabilities, manage objects, permissions, and release workflows; integrate Snowpark logic and CI/CD.
• Define and track data and model quality KPIs (precision, recall, coverage, drift, errors).
• Integrate Arize or comparable platform for observability and root-cause analysis of model behavior.
• Support production rollout: deployment automation, runbooks, documentation, and handoff to internal teams.

Skills

Snowflake expertise

AWS

Generative AI apps

SQL optimization

Snowflake Cortex AI

Snowflake Native Apps

CI/CD (GitHub Actions/Jenkins)

IaC (Terraform/CloudFormation)

Structured text extraction pipelines

ML/LLM observability (Arize)

Python/Scala (Snowpark)

Data quality and KPI tracking

Stakeholder communication

Education

Bachelor's degree in CS/Engineering/Data Science

AWS/Snowflake certifications

Tools

Snowflake

AWS

Snowpark (Python/Scala)

GitHub Actions

Jenkins

Terraform

CloudFormation

Arize

Job description

Location: remote work (Base location Chicago, IL)

Duration: 6+ Months

PURPOSE:

The Senior AI Data Engineer will join Data Engineering team to deliver data and AI capabilities for key initiatives. This role partners closely with Data Product Management, and Data Science to design, build, and operationalize scalable ingestion, transformation pipelines, GenAI-enabled pipelines in Snowflake and AWS.

POSITION RESPONSIBILITIES:

• Design and implement ingestion pipelines from multiple source systems into Snowflake using AWS services and Python.
• Build and optimize SQL-based transformation pipelines for free-form text cleansing, normalization, and deduplication.
• Implement Snowflake Cortex AI workflows for classification, extraction, and structured data generation.
• Develop robust prompt versioning, response parsing, validation, and fallback logic.
• Snowflake Native App Development : Build and deploy Snowflake Native Apps to package data/AI capabilities; manage application objects, permissions, versioning, upgrades, and release workflows; integrate Snowpark-based logic and CI/CD for repeatable delivery.
• Define, instrument, and track data and model quality KPIs, including precision, recall, coverage, drift, and error rates.
• Integrate Arize (or a comparable platform) for observability, monitoring, and root-cause analysis of model behavior.
• Support production rollout and operational readiness: deployment automation, runbooks, documentation, and handoff to internal teams.

EXPERIENCE AND QUALIFICATIONS:

• 8+ years of experience with 5+ years of Data Engineering experience
• 2+ years of hands-on experience developing and deploying Generative AI applications.
• Strong Snowflake expertise, including advanced SQL, Tasks/Streams, stored procedures, and performance tuning.
• Hands-on experience with Snowflake Cortex AI (or an equivalent production LLM platform).
• Experience developing Snowflake Native Apps using the Snowflake Native App Framework (e.g., Snowpark with Python/Scala, application packaging, versioning, upgrades, and secure distribution patterns).
• Strong AWS experience, including S3, Lambda, Glue, Step Functions, DynamoDB, EC2, CloudFormation, and Kinesis.
• Experience with CI/CD tools such as GitHub Actions and/or Jenkins; and IaC with Terraform and/or CloudFormation.
• Experience building structured extraction pipelines from free-text inputs at scale (parsing, validation, deduplication, and quality controls).
• Experience implementing ML/LLM observability with Arize (or a comparable platform).
• Strong understanding of evaluation design, error analysis, and drift monitoring.
• Ability to work independently and deliver against milestone-based outcomes.
• Strong communication skills with technical and non-technical stakeholders.

PREFERRED SKILLS:

• Recent and demonstrable experience automating windows tasks with relevant scripting
• Experience in hospitality, customer profile, personalization, or preference management domains.
• Experience with human-in-the-loop workflows and taxonomy-driven classification systems.
• Familiarity with data privacy and compliance requirements for customer data.
• Experience with Snowflake governance and security patterns (e.g., RBAC, masking policies, row access policies, tagging).
• Experience with automated regression testing for prompts and LLM outputs (prompt evaluation frameworks).
• Experience with Snowpark Container Services or containerized ML/AI services operating close to data.

EDUCATION:

• Bachelor’s degree in Computer Science, Engineering, Data Science, or equivalent practical experience.
• Relevant certifications are a plus (AWS, Snowflake).