Databricks Data Engineer (Hybrid) - Spark & Azure

Peak Performers; Austin (TX); Hybrid; USD 120,000 - 150,000; Full time

Benefits offered by this job

Health insurance

403(b) retirement plan

Weekly direct deposit

Job summary

Peak Performers partners with a Texas State Agency to hire a hybrid Data Engineer in Austin, TX. The role focuses on building scalable data pipelines with Apache Spark on Databricks and integrating Databricks with Azure Data Factory for analytics, ML, and governance.

Key duties include ETL/ELT design, performance optimization, data quality checks, and collaboration with data scientists and stakeholders in an agile environment. W2 contract with competitive hourly pay.

Qualifications

• 4+ years ETL/ELT for structured and unstructured data.
• 4+ years CI/CD, DevOps, and version control experience.
• 4+ years collaborating with data scientists, analysts, stakeholders in agile settings.
• 4+ years designing and maintaining data models, schemas, and DB structures.
• 4+ years evaluating data storage solutions including Azure Data Lake Storage and data warehouses.
• 4+ years data validation, metadata, lineage, and cataloging practices.
• 4+ years implementing data security (encryption, access control, auditing) and regulatory compliance.
• Proficiency in Python and R; strong SQL querying and data manipulation.
• 4+ years on Azure cloud platform; strong debugging skills.
• 3+ years designing scalable data pipelines with Spark on Databricks; Databricks notebooks

Responsibilities

• Designs and develops scalable data pipelines using Spark on Databricks.
• Implements ETL/ELT workflows for structured and unstructured data.
• Optimizes Spark jobs for performance and cost efficiency.
• Integrates Databricks with cloud services like Azure Data Factory.
• Automates deployments using CI/CD tools and version control.
• Collaborates with data scientists, analysts, and stakeholders.
• Ensures data quality, governance, and security using Unity Catalog or Delta Lake.
• Designs and maintains data models, schemas, and database structures.
• Evaluates data storage solutions including data lakes and data warehouses.
• Performs data validation, quality checks, and metadata management.
• Contributes to data governance initiatives and data cataloging.
• Implements data security measures, including encryption and access controls, with regulatory compliance.

Skills

ETL/ELT workflows

CI/CD tools

Cross-functional collaboration

Data modeling

Azure Data Lake Storage

Data governance

Data security

Python

R

SQL querying

Azure cloud

Troubleshooting

Databricks

Spark

ML libraries

Databricks notebooks

Tools

Databricks

Azure Data Factory

Git

CI/CD pipelines

Unity Catalog

Delta Lake

Azure

MLflow

Spark UI

Job description

Peak Performers partners with a Texas State Agency to hire a hybrid Data Engineer in Austin, TX. The role focuses on building scalable data pipelines with Apache Spark on Databricks and integrating Databricks with Azure Data Factory for analytics, ML, and governance.

Key duties include ETL/ELT design, performance optimization, data quality checks, and collaboration with data scientists and stakeholders in an agile environment. W2 contract with competitive hourly pay.