Senior Data Engineer: Scalable ETL, Spark & Cloud

Machine Learning Technologies, LLC; Charlotte (NC); Presencial; USD 120.000 - 180.000; Tempo integral

Resumo da vaga

Machine Learning Technologies, LLC in Charlotte, NC, seeks an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics infrastructure. You will work with Python, PySpark, SQL, and cloud-based platforms to deliver reliable data products for stakeholders.

You will implement ETL/ELT pipelines, data lake and data warehouse solutions, and collaborate with analysts and architects to ensure data quality, governance, and scalable performance across the data

Qualificações

• Experience designing and building ETL/ELT pipelines into data lakes and warehouses.
• Strong Python and PySpark development skills.
• Advanced SQL and performance tuning for large datasets.
• Hands-on Spark and Databricks experience in cloud environments.
• Experience with AWS services (S3, Glue, Redshift, Lambda).

Responsabilidades

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows using Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Apache Spark and Databricks.
• Integrate data from multiple sources including APIs and DBs.
• Develop and maintain data models for analytical reporting.
• Implement data quality checks and validation processes.
• Optimize query performance and pipeline execution.
• Collaborate with data analysts, architects, and business stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Conhecimentos

Python

PySpark

SQL

Airflow

Data modeling

Agile

CI/CD

Git

Data warehousing

Cloud platforms

Ferramentas

Apache Spark

Databricks

Snowflake

Redshift

Kafka

Terraform

Docker

Delta Lake

Descrição da vaga

Machine Learning Technologies, LLC in Charlotte, NC, seeks an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics infrastructure. You will work with Python, PySpark, SQL, and cloud-based platforms to deliver reliable data products for stakeholders.

You will implement ETL/ELT pipelines, data lake and data warehouse solutions, and collaborate with analysts and architects to ensure data quality, governance, and scalable performance across the data