data engineer

Intone Inc; Seattle, Northern (WA, KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Intone Inc. is seeking a Senior Data Engineer for a 6-month contract in Seattle, WA. The role centers on designing, developing, optimizing, and maintaining scalable data platforms and pipelines in a cloud-based environment.

You will design and implement ETL/ELT pipelines with Python, PySpark, Spark SQL, and SQL, and build batch and streaming solutions using Spark, Kafka, and Flink. Cloud work spans AWS and/or GCP with services like S3, EMR, Glue, BigQuery, and Dataproc.

Qualifications

• 4+ years designing, developing, and maintaining scalable data platforms and pipelines.
• Hands-on Python and PySpark experience with Spark SQL.
• Proficient in Airflow for orchestrating ETL/ELT workflows.
• Experience on AWS and/or GCP cloud environments.
• Familiar with Data Lakehouse architectures and data warehousing concepts.
• Strong background in batch and streaming data processing.
• Experience with cloud-based data platforms and large-scale transformations.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and Spark.
• Build batch and streaming solutions with Spark, Kafka, Flink as applicable.
• Develop cloud data solutions across AWS and/or GCP.
• Work with AWS: S3, EMR, Glue, Athena, Redshift; and GCP: Dataproc, BigQuery, Cloud SQL.
• Develop and manage Airflow DAGs, scheduling, retries, and monitoring.
• Implement Medallion/Lakehouse architectures and data lakehouse tech.
• Optimize pipelines for performance, cost, and reliability; ensure data quality and governance.

Skills

Python

PySpark

Spark SQL

Airflow

AWS/GCP

Data Lakehouse

ETL/ELT

Batch/Streaming

Databricks

SQL

Kubernetes/Docker

BigQuery/Athena/Redshift

Tools

Databricks

Delta Lake

Snowflake

BigQuery

Trino

Iceberg

Kafka

Flink

Job description

We are seeking a Senior Data Engineer for a 6-month contract position based in Seattle, Washington. This role focuses on designing, developing, optimizing, and maintaining scalable data platforms and pipelines in a cloud-based environment.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, Spark SQL, and SQL.
• Build high-performance batch and streaming data processing solutions using Apache Spark and technologies such as Kafka/Flink.
• Develop cloud-based data solutions across AWS and/or GCP.
• Work with AWS services including S3, EMR, Glue, Athena, Redshift, Lambda, and CloudWatch.
• Work with GCP services including Dataproc, GCS, BigQuery, and Cloud SQL.
• Develop and manage workflows using Apache Airflow, including DAG development, scheduling, monitoring, retries, and failure handling.
• Design and implement Medallion/Lakehouse architectures using Bronze, Silver, and Gold data layers.
• Work with Databricks, Delta Lake, Iceberg, Parquet, Trino, or equivalent data lakehouse technologies.
• Develop incremental processing, CDC, MERGE/UPSERT, schema evolution, and SCD Type I/II implementations.
• Design dimensional models involving fact and dimension tables, Star Schema, and Snowflake Schema.
• Build reliable data ingestion frameworks from databases, APIs, files, SaaS applications, and streaming sources.
• Optimize Spark workloads using partitioning, partition pruning, broadcast joins, AQE, caching, Z-Ordering, and shuffle optimization.
• Troubleshoot and optimize production pipelines for performance, scalability, SLA adherence, and cloud cost efficiency.
• Implement automated data quality, validation, reconciliation, monitoring, and alerting frameworks.
• Participate in cloud/platform migrations and perform data validation and production-readiness testing.
• Develop CI/CD and deployment workflows using Git, Jenkins, GitHub Actions, Docker, and Kubernetes.
• Collaborate with analytics, backend, product, and business teams to deliver reliable data products.
• Ensure data security, governance, lineage, compliance, and operational reliability across data platforms.

Qualifications

• Required
  • 4+ years of experience in designing, developing, optimizing, and maintaining scalable data platforms and pipelines.
  • Strong hands-on experience with Python and PySpark.
  • Proficiency in Apache Spark and SQL.
  • Experience with Airflow.
  • Experience with AWS or GCP (or both).
  • Familiarity with Data Lakehouse architectures.
  • Experience with ETL/ELT development.
  • Experience with Data Warehousing concepts.
  • Comfort working with both batch and streaming data processing.
  • Experience with cloud-based data platforms.
  • Experience with large-scale data transformations.
  • Experience with data migration projects.
  • Knowledge of data quality principles and practices.
  • Experience optimizing production data pipelines.
• Preferred
  • Experience with Databricks.
  • Experience with Delta Lake.
  • Experience with Kafka or Flink.
  • Experience with Snowflake.
  • Experience with BigQuery.
  • Experience with Trino.
  • Experience with Iceberg.