AI-Driven DataOps Engineer — Azure & Databricks (Remote)

Greystar; Charleston (SC); Remote; USD 120,000 - 150,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Disability insurance

Life insurance

Paid time off

Sabbatical program

401(k) with company match

Parental leave

Job summary

Greystar, a leading real estate platform, seeks a DataOps Engineer to own Databricks-based data pipelines and Azure integration. You will implement AI-driven observability, manage the full deployment lifecycle, and enforce platform standards while optimizing costs and reliability.

You will collaborate with cross-functional teams to deliver scalable, self-healing data infrastructure and advocate for AI-first DataOps practices across the DMP stack.

Qualifications

• 7+ years of DataOps, data engineering, or platform engineering in a production environment.
• Expert-level hands-on experience with Databricks: Delta Live Tables, Jobs/Workflows, Unity Catalog, Spark tuning.
• Strong command of the Azure data services ecosystem: ADF, ADLS Gen2, Azure Monitor, Log Analytics, Key Vault.
• Demonstrated production use of AI tools in DataOps or data observability workflows — LLM-assisted diagnostics, intelligent alerting, etc.
• Proven CI/CD experience using GitHub Enterprise — branch strategies, PR automation, release management for data pipelines.
• Solid Python and/or Scala skills for pipeline development; SQL fluency for Gold layer transformation and DQ validation.
• Experience with medallion / lakehouse architecture patterns and multi-environment deployment discipline.

Responsibilities

• Own the design, build, and optimization of data pipelines on Databricks using Delta Live Tables, PySpark, Workflows, and Jobs across the full DMP medallion stack.
• Administer and govern the Databricks workspace: Unity Catalog, cluster policies, access controls, compute configurations, and Delta table lifecycle management.
• Tune Spark jobs for performance, reliability, and cost — profiling bottlenecks, optimizing partitioning, managing Z-ordering, and controlling compute spend.
• Leverage Databricks Mosaic AI and Genie to build AI-native DataOps capabilities including intelligent pipeline monitoring, anomaly detection, and natural language data access.
• Architect and enforce DMP platform standards: naming conventions, schema evolution policies, SLA tiers, and medallion layer contracts.
• Own the full deployment pipeline for DMP data workflows — promoting changes from development through staging to production with rigor and minimal disruption.

Skills

Databricks

Azure

Python

SQL

CI/CD GitHub

Delta Live Tables

DataOps

AI in DataOps

Tools

GitHub Enterprise

Linear

Azure Monitor

Databricks Mosaic AI

Genie

Job description

Greystar, a leading real estate platform, seeks a DataOps Engineer to own Databricks-based data pipelines and Azure integration. You will implement AI-driven observability, manage the full deployment lifecycle, and enforce platform standards while optimizing costs and reliability.

You will collaborate with cross-functional teams to deliver scalable, self-healing data infrastructure and advocate for AI-first DataOps practices across the DMP stack.