Lead Data Engineer

Quantum Technologies. LLC; Fort Mill (SC); Presencial; USD 110.000 - 160.000; Jornada completa

Descripción de la vacante

Quantum Technologies LLC seeks a data engineering professional to analyze and modernize legacy ETL processes, translating them into scalable ETL/ELT solutions in Fort Mill, SC. You will design and optimize PySpark and Scala-based Spark jobs to support robust data pipelines.

Responsibilities include supporting Sqoop, managing AWS Glue jobs and Data Catalog, applying Protegrity for data masking, and leading data migration initiatives with AWS DataSync.

Formación

• Experience designing and modernizing ETL/ELT pipelines.
• Strong PySpark and Scala-based Spark development.
• Experience with AWS Glue jobs, crawlers, and workflows.
• Familiarity with Sqoop, JDBC configurations, and incremental loads.
• Ability to optimize data processing with partitioning and parallelism.
• Experience with data masking/tokenization (Protegrity) in ETL.
• Schema evolution handling and Glue Data Catalog management.
• Unit and integration testing for data pipelines.
• Lead data migration initiatives, including S3-to-S3 migrations with AWS DataSync.
• Source system analysis and schema/relationship assessment.

Responsabilidades

• Analyze and modernize legacy ETL processes and translate them into scalable ETL/ELT solutions.
• Design, develop, and optimize PySpark and Scala-based Spark jobs.
• Interpret and support Sqoop jobs, including JDBC configurations, incremental load logic, and syntax.
• Develop and manage AWS Glue jobs, crawlers, and workflows.
• Orchestrate Glue jobs using triggers, workflows, and scheduling mechanisms.
• Implement complex data transformation logic including joins, filters, aggregations, cleansing, and formatting.
• Apply Protegrity integration for data masking and tokenization of sensitive data during ETL processing.
• Optimize Glue job performance using partitioning strategies, parallel execution, and resource tuning.
• Handle schema evolution and manage changes in source data structures.
• Manage and maintain Glue Data Catalog, including metadata, table definitions, and schema versioning.
• Perform unit testing and integration testing for data pipelines.
• Lead data migration initiatives, including S3 to S3 migrations using AWS DataSync.
• Conduct source system analysis, including schema, data types, dependencies, and relationships.
• Design and document system architecture for individual components and the overall data pipeline.
• Produce and maintain technical documentation and manage development lifecycle.
• Support deployed code in production environments and troubleshoot issues as needed.

Conocimientos

PySpark

Scala

Spark

Sqoop

Protegrity integration

Partitioning strategies

Schema evolution

AWS Glue

Data transformation

Data Catalog

Herramientas

PySpark

Scala

Spark

Sqoop

AWS Glue

Glue Data Catalog

AWS DataSync

Protegrity

JDBC configurations

DataSync

Descripción del empleo

• Analyze and modernizelegacy ETL processesand translate them into scalableETL/ELT solutions.
• Design, develop, and optimizePySpark and Scala-based Spark jobs.
• Interpret and supportSqoop jobs, including JDBC configurations, incremental load logic, and syntax.
• Develop and manageAWS Glue jobs, crawlers, and workflows.
• Orchestrate Glue jobs usingtriggers, workflows, and scheduling mechanisms.
• Implement complexdata transformation logicincluding joins, filters, aggregations, cleansing, and formatting.
• ApplyProtegrity integrationfor data masking and tokenization of sensitive data during ETL processing.
• Optimize Glue job performance usingpartitioning strategies, parallel execution, and resource tuning.
• Handleschema evolutionand manage changes in source data structures.
• Manage and maintainGlue Data Catalog, including metadata, table definitions, and schema versioning.
• Performunit testing and integration testingfor data pipelines.
• Leaddata migration initiatives, includingS3 to S3 migrations using AWS DataSync.
• Conductsource system analysis, including schema, data types, dependencies, and relationships.
• Design and documentsystem architecturefor individual components and the overall data pipeline.
• Produce and maintaintechnical documentationand manage development lifecycle.
• Support deployed code inproduction environmentsand troubleshoot issues as needed.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.