Senior Data Engineer

Ethereum Technologies; Atlanta (GA); On-site; USD 85,000 - 116,000; Full time

Job summary

Ethereum Technologies is seeking a Senior Data Engineer in Atlanta for an 8‑month engagement. You will design production-grade PySpark pipelines, manage Snowflake environments, and enable Cortex AI capabilities to deliver scalable data products.

Join a team that works with Snowflake, AWS, and modern data lakehouse patterns to optimize performance, cost, and reliability while collaborating with architects, analysts, TPMs and business stakeholders.

Qualifications

• 6+ years of enterprise data engineering experience with PySpark, Spark SQL, Python and advanced SQL.
• Production experience reading large S3 Parquet/Delta datasets and optimizing PySpark processing.
• Deep Snowflake experience: data modeling, SnowSQL, Snowpipe, Streams, Tasks, RBAC, and Iceberg Tables.
• Exposure to Snowflake Cortex AI, semantic analytics, and Snowpark Python; understanding GenAI/LLMs/RAG.
• Hands-on AWS services: S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, Redshift; orchestration tools like Airflow.
• Knowledge of data warehousing, ETL/ELT, data lakes and lakehouse patterns; strong communication and collaboration skills.

Responsibilities

• Design production-grade PySpark batch/streaming pipelines reading S3 Parquet/Delta data.
• Tune Spark performance, partitions, caching, joins, and cost optimization.
• Engineer Snowflake warehouses, schemas, SnowSQL, Snowpipe, Streams, Tasks, and Materialized Views.
• Create and manage Iceberg tables; implement incremental ingestion and CDC patterns.
• Optimize Snowflake with profiling, clustering, and micro-partition pruning.
• Develop Snowpark Python DataFrame transformations, UDFs, and stored procedures.
• Integrate Cortex Analyst/Search/AI Functions and Cortex AI workflows.
• Build AWS-native solutions using S3, Glue, EMR, Lambda, and Redshift; ensure security and monitoring.
• Modernize ETL tools; ensure data quality, lineage, and runbook documentation.
• Collaborate with architects, analysts, TPMs and business teams to deliver secure, cost-effective data products.

Skills

PySpark

Python

Spark SQL

SQL

ETL/ELT

Data warehousing

Tools

Snowflake

SnowSQL

Snowpipe

Iceberg Tables

RBAC

AWS

S3

Glue

EMR

Airflow

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title: Senior Data Engineer

Location: Atlanta, GA

Duration: 8 Months

Bill Rate: $73/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description

• Design production-grade PySpark batch/streaming pipelines; read S3 Parquet/Delta datasets, implement complex transformations and high-volume deduplication.
• Tune Spark using partitioning, caching, broadcast joins, shuffle/skew optimization and execution-plan analysis; balance speed, reliability and cloud cost.
• Engineer Snowflake warehouses, schemas and models; implement SnowSQL, Snowpipe, Snowpipe Streaming, Streams, Tasks, Dynamic Tables and Materialized Views.
• Create Snowflake-managed or externally managed Apache Iceberg tables for interoperable lakehouse solutions; implement incremental ingestion and CDC patterns.
• Optimize Snowflake with query profiling, clustering, micro-partition pruning, warehouse sizing, caching and materialization strategies.
• Build Snowpark Python DataFrame transformations, UDFs and stored procedures; evaluate Snowpark Container Services for advanced processing.
• Enable Snowflake Cortex Analyst (semantic models/views, natural-language-to-SQL), Cortex Search (semantic retrieval/RAG) and Cortex AI Functions (summarization, classification, extraction).
• Explore Cortex Agents, Snowflake Intelligence and Snowflake Copilot; integrate LLM workflows, embeddings, APIs, prompt evaluation and governed AI access as appropriate.
• Build AWS-native solutions using S3, Glue, EMR, Lambda, Step Functions and Redshift; manage IAM, CloudWatch, orchestration and operational monitoring.
• Modernize DataStage/Informatica ETL; ensure data quality, validation, reconciliation, lineage, security, audit readiness and complete technical/runbook documentation.
• Collaborate with architects, analysts, TPMs and business teams to deliver maintainable, secure and cost-effective data products.

Required Qualifications

• 6+ years of enterprise data engineering; strong hands-on PySpark, Spark SQL, Python and advanced SQL (CTEs, window functions, joins, subqueries, tuning).
• Production experience reading large S3 Parquet/Delta datasets, implementing deduplication, and optimizing high-volume PySpark processing.
• Deep Snowflake experience: data modeling, SnowSQL, Snowpipe, Streams, Tasks, RBAC, Iceberg Tables, Dynamic Tables, Materialized Views and performance/cost tuning.
• Working exposure to Snowflake Cortex AI (Cortex Analyst, Search or AI Functions), semantic analytics and Snowpark Python; understanding of GenAI, LLMs, embeddings and RAG.
• Hands-on AWS S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch and Redshift; Airflow, Step Functions or comparable orchestration.
• Knowledge of data warehousing, ETL/ELT, data lakes and lakehouse patterns; strong communication, problem-solving and stakeholder collaboration.

Preferred Qualifications

• Hands-on Cortex Analyst semantic models/views, Cortex Search/RAG, Cortex Agents, Snowflake Intelligence, Copilot, Snowpark Container Services, Streamlit in Snowflake or Native Apps.
• IBM DataStage/Informatica modernization; Git, Jenkins, GitHub Actions, Terraform and CI/CD; Great Expectations/dbt tests; Kafka/Kinesis; Databricks; Control-M and shell scripting.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.