Data Engineer

Robotic Technologies Automation; Houston (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Robotic Technologies Automation seeks a skilled Databricks/Big Data Engineer to develop and optimize Spark jobs and data pipelines on Azure Databricks. You will design lakehouse solutions, implement streaming transformations, and ensure reliable operation of live Databricks workloads.

Ideal candidates have expertise in PySpark/Scala, SQL performance tuning, and container orchestration with Docker/Kubernetes. Prior DevOps experience is a plus.

Qualifications

• Experience with Apache Spark and Databricks platforms.
• Develop and optimize Spark jobs and data pipelines.
• Design data warehouse/lakehouse solutions using Databricks.
• Ability to monitor and debug live Databricks jobs.
• Knowledge of Azure Databricks and data processing at scale.

Responsibilities

• Develop and optimize Spark jobs and data transformations.
• Implement streaming and batch data pipelines with Databricks.
• Design and implement data warehouse/lakehouse solutions.
• Monitor, debug, and resolve issues in live Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Build ingestion, streaming, and transformation solutions.

Skills

Databricks

Apache Spark

Spark DataFrames

Spark SQL

Azure Databricks

Databricks Job Orchestration

Spark Structured Streaming

Data pipelines

Dataset and DataFrame

Cluster sizing

Serverless Databricks

DevOps for Databricks

Python/Scala

Docker

Kubernetes

Tools

Docker

Kubernetes

Azure

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

Mandatory Technical Skills:

• Apache Spark, Spark DataFrames, Notebooks and Spark SQL
• ETL/ELT and data pipeline development
• Scala/PySpark and SQL performance tuning
• Azure Databricks and large-volume data processing
• Databricks Expertise
• Unity Catalog
• Auto Loader
• Lakehouse and Medallion Architecture
• Databricks Serverless Options
• Dataset and DataFrame
• Cluster Configuration and Sizing
• Databricks Job Orchestration
• Spark Structured Streaming

Additional Responsibilities & Requirements:

• Develop and optimize Spark jobs, data transformations, and processing workflows.
• Implement complex data streaming and transformation solutions using Azure Databricks.
• Design and implement data warehouse/lakehouse solutions.
• Monitor, debug, and resolve issues in live Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Build data ingestion, streaming, and transformation solutions.
• Experience with Apache Big Data technologies and languages such as PySpark, R, or Java is advantageous.
• Docker and Kubernetes experience is good to have.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.