Senior Databricks Data Engineer: Scalable ETL & Analytics

Machine Learning Technologies, LLC; Dallas (TX); On-site; USD 150,000 - 190,000; Full time

Job summary

Machine Learning Technologies, LLC seeks a Senior Databricks Data Engineer to design and optimize large-scale data pipelines on the Databricks platform in a Dallas-area setting. You will build end-to-end ETL/ELT processes, integrate with AWS S3, Snowflake, Airflow, and Glue, and deliver enterprise-grade analytics capabilities.

The role emphasizes scalable data architectures, data quality, governance, and collaboration with analytics teams to support regulatory and business insights.

Qualifications

• 7+ years hands-on experience with Databricks and modern data engineering platforms.
• Strong expertise in PySpark, Spark SQL and Apache Spark.
• Experience building ETL/ELT pipelines on cloud platforms.
• Experience with AWS S3, Snowflake, Apache Airflow and AWS Glue.
• Knowledge of data modeling, data warehousing, and big data processing.
• Experience with JSON, Parquet, Avro and CSV formats.
• Hands-on data quality frameworks, monitoring and governance.

Responsibilities

• Design, develop, and maintain scalable data solutions on Databricks using PySpark and Spark SQL.
• Build end-to-end ETL/ELT pipelines integrating Databricks with AWS services like S3, Glue, and Snowflake.
• Transform large-scale structured and semi-structured data for reporting and regulatory needs.
• Develop reusable data processing frameworks for ingestion, cleansing, validation, and enrichment.
• Tune Spark applications and Databricks workloads for performance and cost efficiency.
• Collaborate with stakeholders to translate data requirements into scalable platforms.
• Troubleshoot production pipelines and ensure high availability and reliability.
• Support analytics initiatives including segmentation, campaigns, and regulatory reporting.

Skills

Databricks

PySpark

Spark SQL

SQL

Data modeling

Tools

AWS

Airflow

Delta Lake

Unity Catalog

Snowflake

AWS Glue

Job description

Machine Learning Technologies, LLC seeks a Senior Databricks Data Engineer to design and optimize large-scale data pipelines on the Databricks platform in a Dallas-area setting. You will build end-to-end ETL/ELT processes, integrate with AWS S3, Snowflake, Airflow, and Glue, and deliver enterprise-grade analytics capabilities.

The role emphasizes scalable data architectures, data quality, governance, and collaboration with analytics teams to support regulatory and business insights.