Senior Data Engineer: Scalable Pipelines & Spark

Robotics Technologies LLC; Charlotte (NC); On-site; USD 120,000 - 170,000; Full time

Job summary

Robotics Technologies LLC is seeking an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines and engineering solutions. The role focuses on Python, SQL, Spark, Databricks, and cloud platforms.

Responsibilities include building ETL/ELT pipelines, optimizing data workflows with Airflow, and implementing data lake/warehouse architectures. You will collaborate with analysts, architects, and stakeholders on large-scale data initiatives.

Qualifications

• Strong programming experience in Python and PySpark.
• Advanced SQL skills and database optimization.
• Hands-on experience with Apache Spark and Databricks.
• Experience building ETL/ELT pipelines.
• Knowledge of Apache Airflow and workflow orchestration.
• Experience with AWS services such as S3, Glue, Redshift, and Lambda.
• Strong understanding of data warehousing and data lake architecture.
• Experience with Snowflake or Amazon Redshift.
• Familiarity with Git, CI/CD, and Agile methodologies.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build data processing solutions using Python, PySpark, and SQL.
• Develop and optimize data workflows using Apache Airflow.
• Implement data lake and data warehouse solutions.
• Work with large datasets using Apache Spark and Databricks.
• Integrate data from multiple sources, including APIs and databases.
• Develop and maintain data models for analytical reporting.
• Implement data quality checks and validation processes.
• Optimize query performance and pipeline execution.
• Collaborate with data analysts, architects, and business stakeholders.
• Monitor pipeline performance and troubleshoot production issues.

Skills

Python

PySpark

SQL

Apache Spark

Databricks

Airflow

AWS

Snowflake

Redshift

Git

CI/CD

Agile

Terraform

Docker

Tools

Databricks

Airflow

Spark

AWS S3

AWS Glue

AWS Redshift

Lambda

Git

CI/CD tooling

Terraform

Docker

Job description

Robotics Technologies LLC is seeking an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines and engineering solutions. The role focuses on Python, SQL, Spark, Databricks, and cloud platforms.

Responsibilities include building ETL/ELT pipelines, optimizing data workflows with Airflow, and implementing data lake/warehouse architectures. You will collaborate with analysts, architects, and stakeholders on large-scale data initiatives.