Databricks Platform Operations Engineer

Scicominfra; Atlanta (GA); On-site; USD 120,000 - 180,000; Full time

Job summary

Scicom Infrastructure Services is seeking a Databricks Data Operations Specialist to support the stability, reliability, performance, and security of a large-scale Databricks data platform. You will monitor production pipelines, respond to incidents, troubleshoot issues, and coordinate deployments within a multidisciplinary team.

The ideal candidate has hands-on Databricks experience and supports production workloads using Spark, Delta Lake, SQL, Python, cloud infrastructure, monitoring tools,

Qualifications

• Bachelor’s degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience.
• 4+ years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or related technical role.
• At least 2 years of hands-on experience supporting Databricks environments or production Databricks workloads.
• Experience supporting production ETL, ELT, batch, or streaming data pipelines.
• Working knowledge of Databricks, Apache Spark, PySpark or Python, SQL, Delta Lake, Databricks Workflows or Jobs, cloud object storage, data pipeline monitoring, log analysis and troubleshooting.

Responsibilities

• Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services.
• Provide day-to-day operational support for production Databricks environments.
• Monitor scheduled and event-driven data pipelines for failures, delays, data quality issues, and performance degradation.
• Troubleshoot Databricks jobs, Spark workloads, cluster failures, notebook errors, connectivity issues, and data-processing exceptions.
• Restart, rerun, recover, or coordinate remediation of failed workflows according to documented procedures.
• Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads.
• Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes.
• Support configuration and maintenance of Databricks clusters, serverless compute, instance pools, cluster policies, and SQL warehouses.
• Monitor resource utilization and recommend changes to cluster sizing, autoscaling, workload scheduling, and compute policies.
• Help maintain consistent configurations across development, testing, staging, and production environments.
• Support platform upgrades, runtime-version changes, library updates, and configuration changes.
• Coordinate with Databricks support and cloud-service providers when vendor assistance is required.
• Monitor ETL, ELT, streaming, API, and file-based data-integration processes.
• Validate that scheduled data loads complete within required processing windows.
• Investigate missing, late, duplicated, incomplete, or inconsistent data.
• Perform controlled data reprocessing, backfills, reconciliation, and recovery activities.
• Support ingestion and transformation processes involving JSON, CSV, XML, Parquet, APIs, databases, message queues, and cloud storage.
• Monitor Bronze, Silver, and Gold data layers within a medallion architecture.
• Validate source-to-target record counts, control totals, schema consistency, and business-rule compliance.
• Coordinate with data engineers to resolve recurring pipeline or transformation defects.
• Maintain operational runbooks for common failures, recovery procedures, escalation paths, and recurring support tasks.
• Ensure production interventions are documented and performed in accordance with change-control procedures.

Skills

Databricks

Apache Spark

PySpark/Python

SQL

Delta Lake

Databricks Workflows/Jobs

Cloud object storage

Data pipeline monitoring

Log analysis troubleshooting

Education

Bachelor’s degree in CS/IT/Data Eng or related field

Job description

Scicom Infrastructure Services is seeking a Databricks Data Operations Specialist to support the stability, reliability, performance, and security of a large-scale Databricks data platform. You will monitor production pipelines, respond to incidents, troubleshoot issues, and coordinate deployments within a multidisciplinary team.

The ideal candidate has hands-on Databricks experience and supports production workloads using Spark, Delta Lake, SQL, Python, cloud infrastructure, monitoring tools,