Data Engineer

Quantum Technologies. LLC; Charlotte (NC); On-site; USD 120,000 - 170,000; Full time

Job summary

Quantum Technologies. LLC in Charlotte, NC seeks a Senior Data Engineer to design, build, and maintain scalable data pipelines using Python, PySpark, Spark, Databricks, and cloud data platforms.

You will implement ETL/ELT workflows, data lake and warehouse solutions, optimize queries, and collaborate with analysts and stakeholders across teams.

Qualifications

• Experience with Python and PySpark in data engineering.
• Strong SQL skills and database optimization.
• Hands-on Spark and Databricks experience.
• Experience building ETL/ELT pipelines.
• Knowledge of Airflow and workflow orchestration.
• Familiarity with AWS S3, Glue, Redshift, and Lambda.
• Strong data warehousing and data lake architecture understanding.
• Experience with Snowflake or Redshift.
• Familiarity with Git, CI/CD, and Agile methodologies.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows using Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Apache Spark and Databricks.
• Integrate data from multiple sources, including APIs and databases.
• Develop and maintain data models for analytical reporting.
• Implement data quality checks and validation processes.
• Optimize query performance and pipeline execution.
• Collaborate with data analysts, architects, and business stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Skills

Python

PySpark

SQL

Apache Spark

Databricks

ETL/ELT

Airflow

AWS

Data warehousing

Data lake

Snowflake/Redshift

Git/CI-CD

Tools

Terraform

Docker

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Summary:

We are looking for an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines and data engineering solutions. The ideal candidate should have strong expertise in Python, SQL, Apache Spark, Databricks, and cloud-based data platforms.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows using Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Apache Spark and Databricks.
• Integrate data from multiple sources, including APIs and databases.
• Develop and maintain data models for analytical reporting.
• Implement data quality checks and validation processes.
• Optimize query performance and pipeline execution.
• Collaborate with data analysts, architects, and business stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Required Technical Skills

• Strong programming experience in Python and PySpark.
• Advanced SQL skills and database optimization.
• Hands-on experience with Apache Spark and Databricks.
• Experience building ETL/ELT pipelines.
• Knowledge of Apache Airflow and workflow orchestration.
• Experience with AWS services such as S3, Glue, Redshift, and Lambda.
• Strong understanding of data warehousing and data lake architecture.
• Experience with Snowflake or Amazon Redshift.
• Familiarity with Git, CI/CD, and Agile methodologies.

Preferred Skills

• Experience with Kafka or other streaming technologies.
• Knowledge of Delta Lake and data lakehouse architecture.
• Experience with Terraform and Docker.
• Understanding of data governance and data security.
• Exposure to Azure or Google Cloud Platform.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.