Senior Data Engineer

7Th Sky Tech; Palo Alto (CA); Hybrid; USD 170,000 - 210,000; Full time

Job summary

7Th Sky Tech in California seeks a Senior Data Engineer with 8+ years hands-on experience to design, build, and maintain scalable data platforms and pipelines. Core expertise in Python, SQL, ETL/ELT, cloud platforms, data warehousing, and distributed processing is required.

You will collaborate with Data Scientists, Software Engineers, and Analysts to deliver reliable, high‑performance data solutions and enable data-driven decision making.

Qualifications

• 8+ years of professional experience in Data Engineering.
• Strong programming experience with Python.
• Advanced SQL skills and query optimization.
• Hands-on ETL/ELT development and data pipeline design.
• Experience with distributed processing using Spark/PySpark.
• Familiarity with cloud data platforms (AWS, Azure, GCP).

Responsibilities

• Design and maintain scalable ETL/ELT data pipelines.
• Develop data ingestion from databases, APIs, files, and more.
• Architect data warehouses, data lakes, and lakehouse solutions.
• Work with Spark/PySpark for large datasets.
• Implement data quality, monitoring, and error handling.
• Collaborate with cross-functional teams on requirements.
• Mentor junior engineers and contribute to architecture.

Skills

Python

SQL

ETL/ELT

Spark / PySpark

Data Warehousing

Cloud Platforms (AWS/Azure/GCP)

Data Modeling

CI/CD / DevOps basics

Education

Bachelor's degree in CS/Engineering

Master's degree (preferred)

Tools

Databricks

Snowflake

Redshift

Synapse

BigQuery

Airflow

Git / CI/CD

Job description

• Azure – Data Factory, Databricks, Synapse, ADLS

Senior Data Engineer – 8+ Years Experience

Job Type: Full-Time

Work Model: Hybrid

Experience: 8+ Years

Employment: Full-Time

Job Summary

We are looking for an experienced Senior Data Engineer with 8+ years of hands‑on experience designing, developing, and maintaining scalable data platforms and data pipelines. The ideal candidate will have strong expertise in Python, SQL, ETL/ELT, cloud platforms, data warehousing, and distributed data processing.
The candidate will work closely with Data Scientists, Software Engineers, Business Analysts, and other stakeholders to build reliable, high-performance data solutions and enable data-driven decision‑making.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.


• Build and optimize data processing workflows using Python and SQL.


• Develop data ingestion solutions from databases, APIs, files, and other sources.


• Design and maintain enterprise data warehouses, data lakes, and lakehouse architectures.


• Work with large datasets using distributed processing technologies such as Apache Spark/PySpark.


• Develop data models and optimize complex SQL queries for performance.


• Implement data quality, validation, monitoring, and error‑handling frameworks.


• Integrate data from structured and unstructured sources.


• Build and maintain batch and real‑time/streaming data pipelines.


• Work with cloud data platforms such as AWS, Azure, or GCP.


• Implement CI/CD and DevOps practices for data engineering workflows.


• Collaborate with cross‑functional teams to understand business and technical requirements.


• Troubleshoot pipeline failures, performance issues, and data quality problems.


• Ensure data security, governance, scalability, and compliance.


• Mentor junior and mid‑level data engineers and contribute to technical architecture decisions.


• Document data pipelines, architectures, processes, and technical solutions.

Required Skills

• 8+ years of professional experience in Data Engineering or related roles.


• Strong programming experience with Python.


• Advanced SQL skills with experience in query optimization.


• Strong experience with ETL/ELT development and data pipeline architecture.


• Hands‑on experience with Apache Spark / PySpark.


• Experience with cloud platforms such as:
  
  
  
  • AWS – S3, Glue, Redshift, EMR, Lambda
  
  
  • Azure – Data Factory, Databricks, Synapse, ADLS
  
  
  • GCP – BigQuery, Dataflow, Cloud Storage
  
  
  
  


• Strong understanding of data warehousing and dimensional data modeling.


• Experience with relational databases such as SQL Server, PostgreSQL, Oracle, or MySQL.


• Experience with Databricks and/or Snowflake is highly desirable.


• Knowledge of Apache Kafka or other streaming technologies.


• Experience with workflow orchestration tools such as Airflow, Azure Data Factory, or AWS Glue.


• Familiarity with Git, CI/CD, Docker, and DevOps practices.


• Strong understanding of data quality, governance, security, and metadata management.

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.


• Experience designing enterprise-scale data architectures.


• Experience with real‑time data processing and streaming.


• Experience with Terraform or Infrastructure as Code.


• Familiarity with Kubernetes and containerized workloads.


• Experience working in Agile/Scrum environments.


• Strong communication, analytical, and problem‑solving skills.

Technical Environment

Languages: Python, SQL, Scala/Java

Big Data: Apache Spark, PySpark, Hadoop

Cloud: AWS / Azure / GCP

Data Warehousing: Snowflake, Redshift, Synapse, BigQuery

Databases: SQL Server, PostgreSQL, Oracle, MySQL

ETL/ELT: Databricks, AWS Glue, Azure Data Factory

Streaming: Kafka, Spark Streaming

Orchestration: Apache Airflow

DevOps: Git, Jenkins/GitHub Actions, Docker, CI/CD

Methodology: Agile/Scrum

Work Arrangement

This is a Full-Time Hybrid position. The selected candidate will be expected to work both remotely and from the designated office location based on business requirements.

What We’re Looking For

We are seeking a senior‑level engineer who can independently own data engineering projects, contribute to architecture and design decisions, build production‑grade pipelines, and collaborate effectively with technical and business teams.

Skills: data,pipelines,sql,platforms,aws,processing,cloud,etl,azure,apache