Senior Data Engineer - Spark, ETL/ELT & AWS Glue

Quantum Technologies. LLC; Fort Mill (SC); On-site; USD 110,000 - 160,000; Full time

Job summary

Quantum Technologies LLC seeks a data engineering professional to analyze and modernize legacy ETL processes, translating them into scalable ETL/ELT solutions in Fort Mill, SC. You will design and optimize PySpark and Scala-based Spark jobs to support robust data pipelines.

Responsibilities include supporting Sqoop, managing AWS Glue jobs and Data Catalog, applying Protegrity for data masking, and leading data migration initiatives with AWS DataSync.

Qualifications

• Experience designing and modernizing ETL/ELT pipelines.
• Strong PySpark and Scala-based Spark development.
• Experience with AWS Glue jobs, crawlers, and workflows.
• Familiarity with Sqoop, JDBC configurations, and incremental loads.
• Ability to optimize data processing with partitioning and parallelism.
• Experience with data masking/tokenization (Protegrity) in ETL.
• Schema evolution handling and Glue Data Catalog management.
• Unit and integration testing for data pipelines.
• Lead data migration initiatives, including S3-to-S3 migrations with AWS DataSync.
• Source system analysis and schema/relationship assessment.

Responsibilities

• Analyze and modernize legacy ETL processes and translate them into scalable ETL/ELT solutions.
• Design, develop, and optimize PySpark and Scala-based Spark jobs.
• Interpret and support Sqoop jobs, including JDBC configurations, incremental load logic, and syntax.
• Develop and manage AWS Glue jobs, crawlers, and workflows.
• Orchestrate Glue jobs using triggers, workflows, and scheduling mechanisms.
• Implement complex data transformation logic including joins, filters, aggregations, cleansing, and formatting.
• Apply Protegrity integration for data masking and tokenization of sensitive data during ETL processing.
• Optimize Glue job performance using partitioning strategies, parallel execution, and resource tuning.
• Handle schema evolution and manage changes in source data structures.
• Manage and maintain Glue Data Catalog, including metadata, table definitions, and schema versioning.
• Perform unit testing and integration testing for data pipelines.
• Lead data migration initiatives, including S3 to S3 migrations using AWS DataSync.
• Conduct source system analysis, including schema, data types, dependencies, and relationships.
• Design and document system architecture for individual components and the overall data pipeline.
• Produce and maintain technical documentation and manage development lifecycle.
• Support deployed code in production environments and troubleshoot issues as needed.

Skills

PySpark

Scala

Spark

Sqoop

Protegrity integration

Partitioning strategies

Schema evolution

AWS Glue

Data transformation

Data Catalog

Tools

PySpark

Scala

Spark

Sqoop

AWS Glue

Glue Data Catalog

AWS DataSync

Protegrity

JDBC configurations

DataSync

Job description

Quantum Technologies LLC seeks a data engineering professional to analyze and modernize legacy ETL processes, translating them into scalable ETL/ELT solutions in Fort Mill, SC. You will design and optimize PySpark and Scala-based Spark jobs to support robust data pipelines.

Responsibilities include supporting Sqoop, managing AWS Glue jobs and Data Catalog, applying Protegrity for data masking, and leading data migration initiatives with AWS DataSync.