Senior Data Platform Lead Engineer

TechDigital Group; Fremont (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

TechDigital Group is seeking a Senior Data Engineer with over 12 years of experience in designing and modernizing enterprise data platforms. You will architect scalable ETL/ELT pipelines using Python, PySpark, and Azure Data Factory, and lead enterprise-scale Data Architecture initiatives. The role requires proficiency in Advanced SQL and real-time streaming architectures, with a strong focus on data management across various environments including AWS and Azure.

Qualifications

• 12+ years of experience designing and modernizing enterprise data platforms.
• Strong hands-on experience in Advanced SQL, including complex joins and stored procedures.

Responsibilities

• Architected scalable ETL/ELT pipelines using Python and PySpark.
• Led enterprise-scale Data Architecture initiatives.
• Designed and implemented Medallion Lakehouse architectures.
• Engineered large-scale distributed processing workloads.

Skills

Azure Data Factory (ADF)

Azure Databricks & PySpark

Python

Azure Synapse

Azure SQL

Spark SQL

Tools

Apache Spark

AWS Glue

Apache Airflow

Snowflake

Redshift

PostgreSQL

Apache Kafka

Job description

TechDigital Group is seeking a Senior Data Engineer with over 12 years of experience in designing and modernizing enterprise data platforms. You will architect scalable ETL/ELT pipelines using Python, PySpark, and Azure Data Factory, and lead enterprise-scale Data Architecture initiatives. The role requires proficiency in Advanced SQL and real-time streaming architectures, with a strong focus on data management across various environments including AWS and Azure.