Senior Data Engineer: Spark, Databricks & ETL Pipelines

Artifintech; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Artifintech is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and platforms. The ideal candidate excels in Python, SQL, Apache Spark, Databricks, and cloud data services.

You will build ETL/ELT pipelines, optimize workflows with Airflow, and implement data lake and warehouse solutions while collaborating with analysts and stakeholders to ensure high data quality and performance.

Qualifications

• Proficient in Python and PySpark.
• Advanced SQL and database optimization.
• Hands-on Spark and Databricks experience.
• Experience building ETL/ELT pipelines.
• Knowledge of Airflow and workflow orchestration.
• Experience with AWS services (S3, Glue, Redshift, Lambda).
• Strong data warehousing and data lake architecture understanding.
• Experience with Snowflake or Redshift.
• Familiar with Git, CI/CD, and Agile methods.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows with Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Spark and Databricks.
• Integrate data from API and databases.
• Develop data models for analytical reporting.
• Implement data quality checks and validation.
• Optimize query performance and pipeline execution.
• Collaborate with analysts, architects, and stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Skills

Python

PySpark

SQL

Airflow

AWS

Databricks

Spark

Data Warehousing

Redshift

Snowflake

Git

CI/CD

Docker

Terraform

Kafka

Tools

S3

Glue

Lambda

Delta Lake

Azure

GCP

Job description

Artifintech is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and platforms. The ideal candidate excels in Python, SQL, Apache Spark, Databricks, and cloud data services.

You will build ETL/ELT pipelines, optimize workflows with Airflow, and implement data lake and warehouse solutions while collaborating with analysts and stakeholders to ensure high data quality and performance.