Senior Data Engineer: Spark + Snowflake Lakehouse Expert

MACHINE LEARNING TECHNOLOGIES LLC; Atlanta (GA); On-site; USD 90,000 - 117,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Atlanta, GA is seeking a Senior Data Engineer to design and optimize production-grade PySpark pipelines and Snowflake data models.

You will work with streaming/batch processing, Iceberg tables, Cortex AI components, and AWS services to deliver scalable data products while balancing cost, performance, and reliability.

Strong collaboration with architects and business teams is essential to deliver secure, maintainable solutions.

Qualifications

• 6+ years of enterprise data engineering experience.
• Strong hands-on PySpark, Spark SQL, Python and advanced SQL.
• Experience reading large S3 Parquet/Delta datasets and deduplication.
• Snowflake expertise with data modeling, SnowSQL, Snowpipe, Streams, Tasks, RBAC and performance tuning.

Responsibilities

• Design production-grade PySpark batch/streaming pipelines; read S3 Parquet/Delta datasets, implement complex transformations and high-volume deduplication.
• Tune Spark with partitioning, caching, broadcast joins and optimization; balance speed, reliability and cost.
• Engineer Snowflake warehouses, schemas and models; implement SnowSQL, Snowpipe, Streams, Tasks, Dynamic Tables and Materialized Views.
• Create Snowflake-managed or externally managed Iceberg tables; implement incremental ingestion and CDC patterns.
• Optimize Snowflake with profiling, clustering and caching; ensure cost efficiency.
• Build Snowpark Python DataFrame transformations, UDFs and stored procedures; evaluate Snowpark Container Services.

Skills

PySpark

Spark SQL

Python

Advanced SQL

Data engineering

Batch/Streaming

Tools

Snowflake

SnowSQL

Snowpipe

Snowpark

Apache Iceberg

Redshift

S3

Glue

EMR

Lambda

Airflow

Databricks

Kinesis

Job description

MACHINE LEARNING TECHNOLOGIES LLC in Atlanta, GA is seeking a Senior Data Engineer to design and optimize production-grade PySpark pipelines and Snowflake data models.

You will work with streaming/batch processing, Iceberg tables, Cortex AI components, and AWS services to deliver scalable data products while balancing cost, performance, and reliability.

Strong collaboration with architects and business teams is essential to deliver secure, maintainable solutions.