Senior Data Engineer: Spark, Python & Cloud Data Pipelines

Cloud Analytics Technologies, LLC; Charlotte (NC); On-site; USD 110,000 - 160,000; Full time

Job summary

Cloud Analytics Technologies, LLC is seeking an experienced Senior Data Engineer in Charlotte, NC to design, develop, and maintain scalable data pipelines and data engineering solutions. The role emphasizes Python, SQL, Apache Spark, Databricks, and cloud-based platforms.

Responsibilities include building ETL/ELT pipelines, optimizing data workflows with Airflow, and implementing data lake and data warehouse models.

Qualifications

• Strong Python and PySpark programming experience.
• Advanced SQL and database optimization skills.
• Hands-on Apache Spark and Databricks experience.
• Experience building ETL/ELT pipelines.
• Knowledge of Airflow and workflow orchestration.
• Experience with AWS services (S3, Glue, Redshift, Lambda).
• Understanding of data warehousing and data lake architecture.
• Experience with Snowflake or Redshift.
• Familiarity with Git, CI/CD, and Agile methodologies.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows using Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Spark and Databricks.
• Integrate data from multiple sources (APIs and databases).
• Develop and maintain data models for analytical reporting.
• Implement data quality checks and validation processes.
• Optimize query performance and pipeline execution.
• Collaborate with data analysts, architects, and business stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Skills

Python

PySpark

SQL

Apache Spark

Databricks

Airflow

AWS (S3/Glue/Redshift/Lambda)

Data warehousing

Snowflake/Redshift

Git/CI/CD/Agile

Tools

Databricks

Airflow

Snowflake

Redshift

AWS

Git

CI/CD

Terraform

Docker

Job description

Cloud Analytics Technologies, LLC is seeking an experienced Senior Data Engineer in Charlotte, NC to design, develop, and maintain scalable data pipelines and data engineering solutions. The role emphasizes Python, SQL, Apache Spark, Databricks, and cloud-based platforms.

Responsibilities include building ETL/ELT pipelines, optimizing data workflows with Airflow, and implementing data lake and data warehouse models.