AI-Powered Data Pipeline Architect (Databricks + AWS)

United Software Group Inc; Toronto; On-site; CAD 120,000 - 180,000; Full time

Job summary

United Software Group Inc. is seeking a senior Data Engineer to design and build enterprise-scale data pipelines using Databricks, PySpark, and Delta Lake within an AWS data ecosystem. The role emphasizes AI-assisted development, schema inference, data-quality rules, and AI-ready platforms supporting RAG and vector stores.

The candidate will lead complex data engineering initiatives, leverage GitHub Copilot and Claude Code CLI daily, and accelerate pipeline development with AI-powered tooling.

Qualifications

• 7+ years in Data Engineering/Software Engineering building production-grade solutions.
• 4+ years hands-on with Databricks, Delta Lake, Medallion Architecture, PySpark/Spark.
• 4+ years with AWS data ecosystem (S3, EMR, Glue, Lambda, Redshift).
• Experience leading complex Data Engineering initiatives.
• Strong PySpark experience (not Scala-only Spark).
• Demonstrated daily use of GitHub Copilot and/or Claude Code CLI, with concrete productivity examples.
• Experience rapidly building POCs/MVPs using AI-assisted development.

Skills

Data pipelines

Databricks

PySpark

Delta Lake

Medallion architecture

AWS data services

AI-assisted development

GitHub Copilot

Claude Code CLI

Tools

AWS S3

AWS EMR

AWS Glue

AWS Lambda

AWS Redshift

GitHub Copilot

Claude Code CLI

Job description

United Software Group Inc. is seeking a senior Data Engineer to design and build enterprise-scale data pipelines using Databricks, PySpark, and Delta Lake within an AWS data ecosystem. The role emphasizes AI-assisted development, schema inference, data-quality rules, and AI-ready platforms supporting RAG and vector stores.

The candidate will lead complex data engineering initiatives, leverage GitHub Copilot and Claude Code CLI daily, and accelerate pipeline development with AI-powered tooling.