Senior Databricks Data Engineer: Scale Big Data Pipelines

Quantum Technologies. LLC; Dallas (TX); On-site; USD 150,000 - 190,000; Full time

Job summary

Quantum Technologies. LLC in Dallas, TX seeks a Senior Databricks Data Engineer to design, implement, and optimize large-scale data pipelines on the Databricks platform.

You will work with PySpark, Spark SQL, AWS, and Snowflake to deliver scalable analytics, regulatory reporting, and governance-ready data solutions for enterprise needs.

The role emphasizes performance tuning, data quality, and collaboration with stakeholders to ensure reliable, compliant data processing across the organization.

Qualifications

• 7+ years of hands-on Databricks and data engineering experience
• Strong PySpark, Spark SQL, and Apache Spark expertise
• Experience building ETL/ELT pipelines on cloud platforms
• AWS S3, Snowflake, Airflow, and AWS Glue experience
• Knowledge of data modeling, data warehousing, and big data processing
• Experience with JSON, Parquet, Avro, CSV formats
• Data quality frameworks and monitoring skills
• Strong SQL and data analysis skills
• Experience with Delta Lake, Unity Catalog, and Databricks Workflows
• Familiarity with Agile/Scrum delivery

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks using PySpark and Spark SQL.
• Build ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
• Process and transform large-scale structured and semi-structured data for analytics and reporting.
• Create reusable data frameworks for ingestion, cleansing, transformation, validation, enrichment, and governance.
• Tune Spark and Databricks performance for scalability and efficiency.
• Collaborate with stakeholders, architects, and analytics teams to define data requirements.
• Troubleshoot production pipelines ensuring high availability and reliability.
• Support AML/regulatory reporting, customer analytics, and data validation initiatives.

Skills

Databricks

PySpark

Spark SQL

Apache Spark

AWS

AWS S3

Snowflake

Airflow

AWS Glue

Data modeling

Data warehousing

JSON/Parquet/Avro/CSV

Data quality

SQL

Delta Lake

Unity Catalog

Databricks Workflows

DevOps

Agile/Scrum

Tools

Databricks

Snowflake

Airflow

AWS Glue

Unity Catalog

Delta Lake

Databricks Workflows

Job description

Quantum Technologies. LLC in Dallas, TX seeks a Senior Databricks Data Engineer to design, implement, and optimize large-scale data pipelines on the Databricks platform.

You will work with PySpark, Spark SQL, AWS, and Snowflake to deliver scalable analytics, regulatory reporting, and governance-ready data solutions for enterprise needs.

The role emphasizes performance tuning, data quality, and collaboration with stakeholders to ensure reliable, compliant data processing across the organization.