Databricks SME (Subject Matter Expert)

KATBOTZ; United States; Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

Competitive compensation package

Remote work options

Professional development

Job summary

KATBOTZ is actively seeking an experienced Databricks Subject Matter Expert to drive enterprise data engineering and analytics initiatives in a fully remote role. You will design, optimize, and implement scalable Databricks-based data platforms for global clients, working across cross-functional teams to translate requirements into robust data solutions.

The ideal candidate brings 10+ years in data engineering, deep Databricks and Spark expertise, and strong Python/SQL/Scala skills, with

Qualifications

• 10+ years of experience in Data Engineering and Big Data technologies.
• Deep expertise in Databricks and Apache Spark.
• Proficiency in Python, PySpark, SQL and Scala.
• Expertise in Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong understanding of modern lakehouse architecture and data warehousing.
• Experience with cloud platforms (Azure, AWS, GCP).
• Experience integrating Databricks with BI and reporting tools.
• Knowledge of CI/CD pipelines and DevOps practices.

Responsibilities

• Lead design, development, and optimization of Databricks-based data platforms and pipelines.
• Provide technical expertise on Databricks architecture, performance tuning, and best practices.
• Build scalable ETL/ELT workflows using Apache Spark and Databricks.
• Collaborate with stakeholders to gather requirements and deliver solutions.
• Implement Delta Lake, data governance, and data quality frameworks.
• Support migration of legacy data platforms to Databricks.
• Optimize data processing workflows for performance and cost efficiency.
• Mentor teams on Databricks, Spark, and cloud-native solutions.
• Troubleshoot production issues and provide timely resolutions.
• Ensure adherence to security, compliance, and coding standards.

Skills

Databricks

Spark

Python

SQL

Scala

Delta Lake

Unity Catalog

CI/CD

DevOps

Azure AWS GCP

Tools

Databricks Workflows

CI/CD Tools

Delta Lake Utilities

Job description

This is a remote position. KATBOTZ is seeking an experienced Databricks SME to support enterprise-level data engineering and analytics initiatives.

The ideal candidate will have deep expertise in the Databricks ecosystem, big data technologies, cloud platforms, and modern data architecture. The candidate will work closely with cross-functional teams to design, optimize, and implement scalable data solutions for global clients.

Key Responsibilities

• Lead the design, development, and optimization of Databricks-based data platforms and pipelines.
• Provide technical expertise on Databricks architecture, performance tuning, and best practices.
• Build scalable ETL/ELT workflows using Apache Spark and Databricks.
• Collaborate with business stakeholders, architects, and engineering teams to gather requirements and deliver solutions.
• Implement Delta Lake, data governance, and data quality frameworks.
• Support migration of legacy data platforms to Databricks.
• Optimize data processing workflows for performance and cost efficiency.
• Mentor development teams on Databricks, Spark, and cloud-native solutions.
• Troubleshoot production issues and provide timely resolutions.
• Ensure adherence to security, compliance, and coding standards.

Required Skills & Qualifications

• 10+ years of experience in Data Engineering and Big Data technologies.
• Strong hands-on expertise in Databricks and Apache Spark.
• Experience with Python, PySpark, SQL, and Scala.
• Expertise in Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong understanding of data warehousing and modern lakehouse architecture.
• Experience with cloud platforms such as Microsoft Azure, Amazon AWS, or Google GCP.
• Experience integrating Databricks with BI and reporting tools.
• Knowledge of CI/CD pipelines and DevOps practices.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Databricks Certified Professional certification.
• Experience in real-time streaming using Kafka or Structured Streaming.
• Exposure to AI/ML workflows within Databricks.
• Prior experience working with global remote teams.

Benefits

• Competitive compensation package
• Opportunities for professional development and career advancement.
• Flexible working conditions, with remote options available.
• Dynamic and supportive work environment.

Equal Employment Opportunity

KATBOTZ LLC is an Equal Opportunity Employer. We provide equal employment opportunities to all qualified individuals, regardless of race, religion, gender, gender identity, age, marital status, national origin, sexual orientation, citizenship status, veteran status, disability, or any other legally protected status. As an organization, we are unwavering in our commitment to maintaining a discrimination-free work environment, and fostering a culture of inclusivity, belonging and equal opportunity for all employees and applicants.

Originally posted on Himalayas