Databricks Engineer

RevStar; United States; Remote; USD 120,000 - 170,000; Full time

Benefits offered by this job

Remote-first for US-based employees

Paid time off

Medical, dental, vision coverage

401(k) retirement plan

Learning & development stipend

Mentorship & growth opportunities

Exposure to AWS GenAI & cloud tech

Job summary

RevStar is seeking an experienced data engineer to design and optimize scalable ETL/ELT pipelines using Apache Spark and Delta Lake across multi-cloud environments.

You will implement Lakehouse architectures, automate ingestion and data quality workflows, and tune Spark jobs for performance and cost efficiency. Collaboration with data scientists, engineers, and stakeholders is required.

Qualifications

• 3+ years of hands-on data engineering experience in big data and cloud-native architectures.
• 2+ years of Databricks experience with Spark, Delta Lake, and MLflow.
• Databricks Certified Data Engineer Associate or higher.
• Proficiency in Python, SQL, and Spark-based frameworks.
• Experience developing and optimizing large-scale ETL/ELT pipelines.
• Strong understanding of Lakehouse architecture and cloud-agnostic data solutions.
• Familiarity with CI/CD and Infrastructure-as-Code for Databricks, including Terraform and Databricks CLI.
• Knowledge of data governance, security, and compliance (GDPR, HIPAA, or SOC 2).
• Experience in Agile, DevOps, and MLOps environments.
• Additional Databricks certifications, streaming technologies, orchestration tools, AI/ML integration, or client-facing consulting experience preferred.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines with Spark and Delta Lake.
• Implement Lakehouse architectures across multi-cloud environments.
• Automate data ingestion, storage, monitoring, alerting, and data quality workflows.
• Tune Spark jobs for performance, latency, throughput, and cost efficiency.
• Develop CI/CD pipelines and IaC deployments using Terraform and Databricks CLI.
• Build feature engineering pipelines and support model training, tracking, versioning, and deployment with MLflow.
• Operationalize secure, scalable AI/ML models for client applications.
• Enforce data security, access controls, compliance, lineage, metadata management, and documentation standards.
• Collaborate with architects, data scientists, engineers, and client stakeholders to align solutions with business outcomes.

Skills

Python

SQL

Spark

Delta Lake

MLflow

Databricks

Terraform

Databricks CLI

Data governance

CI/CD

Tools

Databricks

Delta Lake

MLflow

Terraform

Databricks CLI

Job description

Description:

• Design, build, and optimize scalable ETL/ELT pipelines using Apache Spark and Delta Lake across multi-cloud environments.
• Implement Lakehouse architectures that process structured and unstructured enterprise data.
• Automate data ingestion, storage, monitoring, alerting, and data quality validation workflows.
• Tune Spark jobs for performance, low latency, high throughput, and cloud cost efficiency.
• Develop CI/CD pipelines and Infrastructure-as-Code deployments using Terraform and Databricks CLI.
• Build feature engineering pipelines and support model training, tracking, versioning, and deployment with MLflow.
• Operationalize secure, scalable AI/ML models for client applications.
• Enforce data security, access controls, compliance, lineage, metadata management, and documentation standards.
• Collaborate with architects, data scientists, engineers, and client stakeholders to align solutions with business outcomes.

Requirements:

• 3+ years of hands-on data engineering experience focused on big data processing and cloud-native architectures.
• 2+ years of hands-on Databricks experience, including Apache Spark, Delta Lake, and MLflow.
• Databricks Certified Data Engineer Associate certification or higher.
• Proficiency in Python, SQL, and Spark-based frameworks.
• Experience developing and optimizing large-scale ETL/ELT pipelines.
• Strong understanding of Lakehouse architecture and cloud-agnostic data solutions.
• Familiarity with CI/CD and Infrastructure-as-Code for Databricks, including Terraform and Databricks CLI.
• Knowledge of data governance, security, and compliance practices, including GDPR, HIPAA, or SOC 2.
• Experience in Agile, DevOps, and MLOps environments.
• Additional Databricks certifications, streaming technologies, orchestration tools, AI/ML integration, or client-facing consulting experience preferred.

Benefits:

• Remote-first work environment for US-based employees.
• Paid time off.
• Medical, dental, and vision coverage for full-time W2 positions.
• Company-sponsored 401(k) retirement plan.
• Annual learning and development stipend for courses, conferences, and certifications.
• Peer mentorship, coaching, and professional growth opportunities.
• Exposure to AWS GenAI, data, and cloud technologies across diverse industries.
• Company outings and volunteer opportunities.