Lead Data Engineer: Scalable Pipelines & Platforms

CNA National; Scottsdale (AZ); On-site; USD 150,000 - 190,000; Full time

Job summary

CNA National, based in Scottsdale, seeks a Senior Data Engineer to design, build, and maintain a modern data platform supporting operational and analytical workloads. You will own end‑to‑end data pipelines, enforce data quality, and collaborate across teams to ensure scalable, secure data solutions.

The role requires a hands‑on leader with 10+ years in data engineering, deep SQL, Python, and experience with Snowflake or Databricks, AWS, and data governance.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or equivalent experience.
• 10+ years of experience in data engineering.
• Mastery of complicated SQL for loading data, including complex joins, subqueries, windowing functions, and common table expressions.
• Advanced proficiency in Python, developing and supporting robust pipelines, frameworks, and automation solutions.
• Proven history designing and managing Data Warehousing solutions including proper loading techniques, star schema modeling, slowly changing dimensions, and aggregate strategies.
• Strong understanding of OLTP modeling (3NF) and how to denormalize for performance or downstream use.
• Production experience with unstructured and semi‑structured data (e.g., JSON, MongoDB, APIs) and integrating it into enterprise data ecosystems.
• Practical expertise implementing and optimizing pipeline orchestration with tools such as Airflow or Prefect.
• Experience implementing and optimizing 3rd party transformation tools such as dbt, Talend, or FiveTran.
• Demonstrated success in delivering scalable solutions using cloud‑based data platforms like Snowflake or Databricks.
• Applied experience leveraging AWS data services (S3, Glue, Redshift, Lambda, Secrets Manager, etc.) in secure and cost‑effective ways.
• Demonstrated ability to ensure compliance with regulatory requirements including PCI‑DSS, GDPR, CPRA, CCPA, and SOX.
• Effective communication skills for engaging both technical and non‑technical audiences and collaborating with business and audit stakeholders.
• Proven capability mentoring peers, promoting best practices, and contributing to team growth.
• High adaptability to changing requirements, emerging technologies, and competing priorities.
• Nice to have: Hands‑on experience with Data Lake and Lakehouse technologies (e.g., Delta Lake, Iceberg, Hudi) for managing large‑scale data assets.

Responsibilities

• Design, build, and maintain scalable data pipelines (ETL/ELT) for batch and streaming use cases.
• Implement data validation and integrity frameworks ensuring accuracy and reconciliation across systems.
• Administer and optimize cloud‑based data services (AWS, Snowflake, Databricks).
• Ensure compliance with data governance and regulations (PCI‑DSS, GDPR, CPRA, SOX).
• Deliver proofs of concepts and lightweight prototypes to validate architectural improvements.
• Collaborate with business units, audit teams, and stakeholders to align data architecture with needs.
• Support reporting and analytics teams by enabling Tableau (or future‑state tools) through well‑structured data models and governance.
• Mentor peers and own projects end‑to‑end to deliver measurable results.
• Diagnose and resolve complex data issues quickly.

Skills

SQL proficiency

Python

Data warehousing

ETL/ELT pipelines

Airflow

dbt

Snowflake

Databricks

AWS

Data governance

Mentoring

Communication

Regulatory compliance

Education

Bachelor’s degree in Computer Science / Information Technology

Tools

Airflow

Prefect

dbt

FiveTran

Snowflake

Databricks

AWS

S3

Redshift

Lambda

Secrets Manager

Job description

CNA National, based in Scottsdale, seeks a Senior Data Engineer to design, build, and maintain a modern data platform supporting operational and analytical workloads. You will own end‑to‑end data pipelines, enforce data quality, and collaborate across teams to ensure scalable, secure data solutions.

The role requires a hands‑on leader with 10+ years in data engineering, deep SQL, Python, and experience with Snowflake or Databricks, AWS, and data governance.