Databricks Data Engineer

Quantum Technologies. LLC; Dallas (TX); Presencial; USD 150.000 - 190.000; Jornada completa

Descripción de la vacante

Quantum Technologies. LLC in Dallas, TX seeks a Senior Databricks Data Engineer to design, implement, and optimize large-scale data pipelines on the Databricks platform.

You will work with PySpark, Spark SQL, AWS, and Snowflake to deliver scalable analytics, regulatory reporting, and governance-ready data solutions for enterprise needs.

The role emphasizes performance tuning, data quality, and collaboration with stakeholders to ensure reliable, compliant data processing across the organization.

Formación

• 7+ years of hands-on Databricks and data engineering experience
• Strong PySpark, Spark SQL, and Apache Spark expertise
• Experience building ETL/ELT pipelines on cloud platforms
• AWS S3, Snowflake, Airflow, and AWS Glue experience
• Knowledge of data modeling, data warehousing, and big data processing
• Experience with JSON, Parquet, Avro, CSV formats
• Data quality frameworks and monitoring skills
• Strong SQL and data analysis skills
• Experience with Delta Lake, Unity Catalog, and Databricks Workflows
• Familiarity with Agile/Scrum delivery

Responsabilidades

• Design, develop, and maintain scalable data pipelines on Databricks using PySpark and Spark SQL.
• Build ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
• Process and transform large-scale structured and semi-structured data for analytics and reporting.
• Create reusable data frameworks for ingestion, cleansing, transformation, validation, enrichment, and governance.
• Tune Spark and Databricks performance for scalability and efficiency.
• Collaborate with stakeholders, architects, and analytics teams to define data requirements.
• Troubleshoot production pipelines ensuring high availability and reliability.
• Support AML/regulatory reporting, customer analytics, and data validation initiatives.

Conocimientos

Databricks

PySpark

Spark SQL

Apache Spark

AWS

AWS S3

Snowflake

Airflow

AWS Glue

Data modeling

Data warehousing

JSON/Parquet/Avro/CSV

Data quality

SQL

Delta Lake

Unity Catalog

Databricks Workflows

DevOps

Agile/Scrum

Herramientas

Databricks

Snowflake

Airflow

AWS Glue

Unity Catalog

Delta Lake

Databricks Workflows

Descripción del empleo

Work Authorization:

We are seeking an experienced Senior Databricks Data Engineer with strong expertise in designing, developing, and optimizing large-scale data engineering solutions on the Databricks platform. The ideal candidate will have extensive hands-on experience in building enterprise-grade data pipelines, processing high-volume datasets, and delivering scalable analytics solutions using PySpark, Spark SQL, and AWS cloud technologies.

Key Responsibilities:

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark.
• Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
• Process and transform large-scale structured and semi-structured datasets to support enterprise reporting, analytics, and regulatory requirements.
• Develop reusable frameworks and components for: Data ingestion Data cleansing and transformation Data validation and reconciliation Data enrichment Business rule implementation Implement robust data quality checks, monitoring, and governance controls to ensure data accuracy and reliability.
• Perform performance tuning and optimization of Spark applications and Databricks workloads to improve scalability and processing efficiency.
• Collaborate with business stakeholders, architects, and analytics teams to understand data requirements and deliver scalable solutions.
• Troubleshoot and resolve production data pipeline issues while ensuring high availability and reliability.
• Support advanced analytics, customer segmentation, campaign analytics, AML/regulatory reporting, and data validation initiatives.

Required Skills:

• Technical Skills 7+ years of hands-on experience with Databricks and modern data engineering platforms.
• Strong expertise in: PySpark Spark SQL Apache Spark Databricks Lakehouse Architecture Experience building and managing ETL/ELT pipelines on cloud platforms.
• Strong experience with: AWS S3 Snowflake Apache Airflow AWS Glue Knowledge of data modeling, data warehousing, and big data processing techniques.
• Experience working with structured and semi-structured data formats such as JSON, Parquet, Avro, and CSV.
• Hands-on experience implementing data quality frameworks and monitoring solutions.
• Strong SQL and data analysis skills.
• Preferred Skills Experience with regulatory reporting, AML, compliance, or financial data domains.
• Exposure to CI/CD and DevOps practices for data engineering.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Knowledge of Agile/Scrum delivery methodologies.
• Key Experience Areas Candidates should have demonstrated experience in one or more of the following: Customer Segmentation Analytics Campaign Analytics AML and Regulatory Reporting Enterprise Data Platforms Production Data Pipeline Support Data Validation and Reconciliation Large-Scale Data Migration and Modernization Programs.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.