Data Lead Engineer

TechDigital Group; Fremont (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

TechDigital Group is seeking a Senior Data Engineer with over 12 years of experience in designing and modernizing enterprise data platforms. You will architect scalable ETL/ELT pipelines using Python, PySpark, and Azure Data Factory, and lead enterprise-scale Data Architecture initiatives. The role requires proficiency in Advanced SQL and real-time streaming architectures, with a strong focus on data management across various environments including AWS and Azure.

Qualifications

• 12+ years of experience designing and modernizing enterprise data platforms.
• Strong hands-on experience in Advanced SQL, including complex joins and stored procedures.

Responsibilities

• Architected scalable ETL/ELT pipelines using Python and PySpark.
• Led enterprise-scale Data Architecture initiatives.
• Designed and implemented Medallion Lakehouse architectures.
• Engineered large-scale distributed processing workloads.

Skills

Azure Data Factory (ADF)

Azure Databricks & PySpark

Python

Azure Synapse

Azure SQL

Spark SQL

Tools

Apache Spark

AWS Glue

Apache Airflow

Snowflake

Redshift

PostgreSQL

Apache Kafka

Job description

Key Skills

Azure Data Factory (ADF), Azure Databricks & PySpark, Azure Synapse, Azure SQL, Python, and Spark SQL

Role Overview

Senior Data Engineer with 12+ years of experience designing and modernizing enterprise data platforms across banking, healthcare and retail domains.

Responsibilities

• Architected scalable ETL/ELT pipelines using Python, PySpark, Databricks, AWS Glue and Azure Data Factory, supporting high-volume transactional and regulatory data processing.
• Led enterprise-scale Data Architecture initiatives, defining logical and physical data models, governance standards and cloud-native platform blueprints across AWS and Azure environments.
• Designed and implemented Medallion (Bronze/Silver/Gold) Lakehouse architectures using Delta Lake, S3, ADLS Gen2, Snowflake, Redshift and Synapse Analytics.
• Engineered large-scale distributed processing workloads using Apache Spark, PySpark, Databricks, EMR, Hive and HDFS, processing billions of records for enterprise analytics.
• Orchestrated complex data workflows using Apache Airflow, Databricks Workflows, AWS Step Functions and Azure Data Factory triggers, ensuring SLA-driven pipeline execution.
• Strong hands-on experience in Advanced SQL, including complex joins, CTEs, window functions, stored procedures, indexing strategies, partitioning and execution plan optimization across Snowflake, PostgreSQL and Oracle.
• Built real-time streaming architectures using Apache Kafka, AWS Kinesis, Azure Event Hub and Service Bus, supporting fraud detection, claims monitoring and operational telemetry.