Databricks SME

Agelix Consulting LLC; Washington (District of Columbia); On-site; USD 120,000 - 150,000; Full time

Job summary

Agelix Consulting LLC is seeking a Databricks Subject Matter Expert (SME) to serve as a technical guide for architecture design and optimization. The ideal candidate will have 8+ years of experience in cloud technology environments and at least 3–5 years dedicated to Databricks.

Responsibilities include optimizing clusters, leading governance processes, and mentoring engineers. Familiarity with AWS and strong proficiency in Python, Scala, or SQL is required. This role is located in Washington, D.C., for 18 months.

Qualifications

• 8+ years of hands-on experience in cloud technology environments.
• 3–5 years of dedicated Databricks experience.
• Proven expertise with Unity Catalog and Delta Lake.

Responsibilities

• Serve as the Databricks SME and guide technical decisions.
• Lead governance processes within Databricks environments.
• Mentor mid-level engineers and contribute to team knowledge sharing.

Skills

Cloud technology environments

Databricks administration

AWS services

Data governance frameworks

Python

Scala

SQL

Tools

Databricks

AWS

Job description

Duration: 18 months

Location: Washington, D.C. 20224

Key Responsibilities

• Serve as the Databricks SME and guide technical decisions for workspace architecture design, configuration, and optimization.
• Lead governance processes within Databricks environments, including access controls, data lineage, and workspace policies.
• Optimize Databricks cluster and job performance; identify and resolve bottlenecks in data pipelines.
• Design and implement secure integration patterns between Databricks and AWS services (e.g., S3, IAM, Glue, Redshift).
• Perform specialized operations including Delta Lake management, Unity Catalog administration, and MLflow tracking.
• Collaborate with government technical leads on architecture reviews and platform enhancement initiatives.
• Develop and maintain technical documentation, runbooks, and best practice guidelines.
• Support platform security requirements and ensure compliance with IRS/Treasury policies.
• Mentor mid-level engineers and contribute to team knowledge sharing.
• Participate in code reviews, design discussions, and agile ceremonies.

Required Qualifications & Skills

• 8+ years of hands‑on experience in cloud technology environments.
• 3–5 years of dedicated Databricks experience including workspace administration and optimization.
• Proven expertise as a Databricks SME with Unity Catalog, Delta Lake, and cluster management.
• Strong proficiency with AWS services including S3, IAM, Glue, Redshift, and VPC networking.
• Experience with data governance frameworks and security controls in cloud environments.
• Proficiency in Python, Scala, or SQL for data engineering tasks.
• Familiarity with federal IT compliance standards (FedRAMP, FISMA) preferred.
• Knowledge of tax‑processing or financial data management systems is highly preferred.
• Databricks Certified Data Engineer Professional certification preferred.
• AWS Certified Data Analytics or Solutions Architect certification preferred.