Senior Big Data Architect – Spark & Cloud Platform

Valiant Technologies LLC; New York (NY); On-site; USD 180,000 - 230,000; Full time

Job summary

Valiant Technologies LLC is seeking a senior data architect/lead data engineer to drive data quality, governance, and platform architecture across cloud data lakes. You will shape frameworks, mentor teams, and partner with leadership to align data initiatives with business goals.

The role requires deep hands-on expertise with Spark, SQL, Python, and a breadth of cloud data services, delivering scalable data solutions with SDLC and DevOps discipline.

Qualifications

• Bachelor's or master's degree in computer science, engineering, or related quantitative field.
• 15+ years of progressive software engineering experience with at least 5+ years in senior data/architecture roles focused on data quality, governance, or architecture.

Responsibilities

• Lead data quality engineering initiatives, define standards for profiling, validation, cleansing, reconciliation, and anomaly detection.
• Architect and deliver large-scale data automation frameworks across cloud platforms and data lakes.

Skills

Apache Spark

Spark SQL/DataFrames

Streaming

Hadoop/HDFS

Delta Lake/Iceberg

Kafka/Kinesis

Cloud data platforms

Advanced SQL

Python

Data quality engineering

Data governance

Data platform architecture

Mentoring teams

SDLC/DevOps/CI/CD

Communication

Education

Bachelor's or Master's in Computer Science or related quantitative field

Tools

AWS (EMR, Glue, Redshift, Lambda, Step Functions, S3)

Azure (Databricks, Synapse Analytics, Data Lake Storage)

Google Cloud (Dataproc, BigQuery, Cloud Storage)

Hadoop ecosystem tools

Job description

Valiant Technologies LLC is seeking a senior data architect/lead data engineer to drive data quality, governance, and platform architecture across cloud data lakes. You will shape frameworks, mentor teams, and partner with leadership to align data initiatives with business goals.

The role requires deep hands-on expertise with Spark, SQL, Python, and a breadth of cloud data services, delivering scalable data solutions with SDLC and DevOps discipline.