Senior Databricks Data Engineer - Spark, AWS & Delta Lake

Robotic Technologies Automation; Dallas (TX); On-site; USD 140,000 - 180,000; Full time

Job summary

Robotic Technologies Automation is seeking a Senior Databricks Data Engineer to design and optimize large-scale data pipelines on the Databricks platform, leveraging PySpark and Spark SQL in an AWS‑based environment.

You will build end-to-end ETL/ELT pipelines, work with S3, Snowflake, Airflow, and Glue, and develop reusable data processing frameworks. The role involves data governance, quality checks, and collaboration with analytics teams.

Qualifications

• 7+ years of hands‑on experience with Databricks and modern data engineering platforms.
• Strong expertise in PySpark, Spark SQL, and Apache Spark.
• Experience in building and managing ETL/ELT pipelines on cloud platforms.
• Proficient with AWS S3, Snowflake, Airflow, and AWS Glue.
• Knowledge of data modeling, data warehousing, and big data processing techniques.
• Experience with JSON, Parquet, Avro, and CSV data formats.
• Hands‑on data quality frameworks, monitoring, and governance.
• Strong SQL and data analysis skills.
• Experience with Delta Lake, Databricks Workflows, Unity Catalog, and Agile delivery.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark.
• Build end-to-end ETL/ELT pipelines with Databricks, AWS S3, Snowflake, Airflow, and AWS Glue.
• Process and transform large-scale structured and semi-structured data for reporting, analytics, and regulatory needs.
• Develop reusable components for ingestion, cleansing, transformation, validation, enrichment, and governance.
• Implement robust data quality checks, monitoring, and governance to ensure data accuracy and reliability.
• Tune Spark applications and Databricks workloads for scalability and performance.
• Collaborate with stakeholders to translate data requirements into scalable solutions.
• Troubleshoot production data pipelines and ensure high availability.
• Support analytics such as customer segmentation, campaign analytics, and regulatory reporting.

Skills

PySpark

Spark SQL

Apache Spark

SQL

Data modeling

Data warehousing

Data quality frameworks

Agile/Scrum

AML/regulatory reporting

Tools

Databricks

AWS S3

Snowflake

Apache Airflow

AWS Glue

Delta Lake

Databricks Workflows

Unity Catalog

Job description

Robotic Technologies Automation is seeking a Senior Databricks Data Engineer to design and optimize large-scale data pipelines on the Databricks platform, leveraging PySpark and Spark SQL in an AWS‑based environment.

You will build end-to-end ETL/ELT pipelines, work with S3, Snowflake, Airflow, and Glue, and develop reusable data processing frameworks. The role involves data governance, quality checks, and collaboration with analytics teams.