Senior Data Engineer: ETL, Spark & Databricks (Charlotte)

MACHINE LEARNING TECHNOLOGIES LLC; Charlotte (NC); On-site; USD 96,000 - 129,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Charlotte, NC is seeking an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines and data engineering solutions. You will work with Python, PySpark, SQL, Apache Spark, Databricks, and cloud platforms to deliver robust data processing.

Responsibilities include building ETL/ELT pipelines, optimizing data workflows with Airflow, developing data models for analytics, implementing data quality checks, and collaborating with

Qualifications

• Strong Python and PySpark proficiency.
• Advanced SQL and database optimization skills.
• Hands-on Spark and Databricks experience.
• Experience building ETL/ELT pipelines.
• Knowledge of Airflow and workflow orchestration.
• Experience with AWS services (S3, Glue, Redshift, Lambda).
• Understanding of data warehousing and lake architectures.
• Experience with Snowflake or Redshift.
• Familiarity with Git, CI/CD, and Agile methodologies.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows using Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Apache Spark and Databricks.
• Integrate data from multiple sources, including APIs and databases.
• Develop and maintain data models for analytical reporting.
• Implement data quality checks and validation processes.
• Optimize query performance and pipeline execution.
• Collaborate with data analysts, architects, and business stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Skills

Python

PySpark

SQL

Apache Spark

Databricks

Airflow

AWS S3

Redshift

Snowflake

Git

CI/CD

Agile

ETL/ELT

Tools

Spark/Databricks

Job description

MACHINE LEARNING TECHNOLOGIES LLC in Charlotte, NC is seeking an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines and data engineering solutions. You will work with Python, PySpark, SQL, Apache Spark, Databricks, and cloud platforms to deliver robust data processing.

Responsibilities include building ETL/ELT pipelines, optimizing data workflows with Airflow, developing data models for analytics, implementing data quality checks, and collaborating with