Databricks Practice Lead / Engineering Manager

scicominfrastructureservices; Atlanta (GA); On-site; USD 180,000 - 230,000; Full time

Job summary

Scicom Infrastructure Services seeks an experienced Databricks Practice Lead / Engineering Manager to provide hands-on technical leadership while managing a team of data engineers, architects, and consultants supporting enterprise and government programs.

The role requires senior Databricks expertise to design modern data platforms, set technical standards, guide delivery, and participate in architecture reviews, troubleshooting, and client-facing solution development.

Qualifications

• Bachelor’s degree or equivalent work experience in computer science, IT, data engineering, or related field.
• 10+ years in data engineering, data architecture, analytics engineering, or related roles.
• 5+ years leading technical engineering teams and delivering Databricks solutions.
• Deep expertise with Databricks Lakehouse Platform, Spark, Delta Lake, and governance frameworks.
• Experience deploying Databricks on Azure, AWS, or Google Cloud.
• Strong CI/CD, Git-based development, automated testing, and infrastructure as code.
• Ability to optimize Spark workloads, cluster configurations, and cloud costs.
• Excellent communication and stakeholder-management skills.

Responsibilities

• Provide hands-on technical leadership for Databricks platforms across enterprise and government programs.
• Design scalable, secure data architectures and establish technical standards.
• Guide delivery teams through architecture reviews, code reviews, and client-facing solution development.
• Mentor engineers, manage resources, and drive performance with clear goals and development plans.
• Oversee platform migrations, modernization efforts, and production support.
• Lead governance, data modeling, ingestion, and data sharing strategies.

Skills

Leadership

Communication

Architecture vision

Mentoring

Problem solving

Client facing

Education

Bachelor’s degree in computer science or related field

Tools

Databricks Lakehouse Platform

Apache Spark / PySpark

Delta Lake

Unity Catalog

Cloud platforms (Azure / AWS / GCP)

CI/CD / IaC

MLflow / MLOps

Databricks Workflows / Jobs

Job description

Position Summary

Scicom Infrastructure Services is seeking an experienced Databricks Practice Lead / Engineering Manager to provide hands‑on technical leadership while managing a team of data engineers, architects, and consultants supporting complex enterprise and government programs.

This role requires a senior Databricks expert who can design and oversee modern data platforms, establish technical standards, guide delivery teams, and remain actively involved in architecture, troubleshooting, code reviews, and client‑facing solution development. The successful candidate will balance deep technical expertise with strong people leadership, delivery management, and stakeholder communication skills.

Key Responsibilities

Databricks Technical Leadership

• Serve as the organization’s subject‑matter expert for the Databricks Lakehouse Platform.
• Design scalable, secure, and highly available data architectures using Databricks, Apache Spark, Delta Lake, and cloud‑native technologies.
• Lead the implementation of batch, streaming, ETL, ELT, analytics, machine‑learning, and AI‑enabled data solutions.
• Define architectural standards for medallion architectures, data modeling, ingestion, transformation, orchestration, and data consumption.
• Establish governance frameworks using Unity Catalog, including data lineage, access controls, auditing, metadata management, and secure data sharing.
• Guide Databricks workspace design, cluster configuration, serverless computing, workload isolation, performance tuning, and cost optimization.
• Oversee integration between Databricks and cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
• Develop or review solutions involving PySpark, Spark SQL, Python, Delta Live Tables, Structured Streaming, Auto Loader, MLflow, and Databricks Workflows.
• Lead platform migrations and modernization efforts from legacy databases, data warehouses, Hadoop environments, and traditional ETL platforms.
• Establish development standards for source control, automated testing, CI/CD, infrastructure as code, monitoring, and production support.
• Conduct architecture reviews, code reviews, technical assessments, and root‑cause analyses.
• Evaluate emerging Databricks capabilities and recommend appropriate adoption strategies.

Team Leadership and Management

• Manage, mentor, and develop a team of Databricks engineers, data engineers, architects, and technical consultants.
• Assign resources and responsibilities based on project needs, employee strengths, availability, and technical complexity.
• Establish measurable goals, performance expectations, development plans, and technical competency standards.
• Conduct regular one‑on‑one meetings, performance reviews, coaching sessions, and technical development activities.
• Support recruiting, interviewing, candidate evaluation, onboarding, and workforce planning.
• Identify technical or performance gaps and coordinate training, mentoring, or corrective action as appropriate.
• Promote collaboration, accountability, documentation, knowledge sharing, and continuous improvement.
• Develop reusable accelerators, reference architectures, templates, and delivery playbooks.
• Build and maintain a strong Databricks practice capable of supporting multiple concurrent client engagements.

Program and Delivery Management

• Provide delivery oversight for Databricks and data‑engineering projects from planning through implementation and operational support.
• Translate business, functional, security, and contractual requirements into technical plans and deliverables.
• Develop project estimates, staffing plans, delivery schedules, milestones, and risk‑mitigation strategies.
• Monitor project scope, schedule, quality, budget, resource utilization, dependencies, and technical risks.
• Ensure deliverables meet client requirements, internal quality standards, security controls, and contractual commitments.
• Coordinate work across engineering, cloud, cybersecurity, data governance, analytics, project‑management, and client teams.
• Track delivery metrics and provide clear status reports to internal leadership, clients, and program stakeholders.
• Lead technical escalations and ensure issues are resolved promptly and appropriately documented.
• Support statements of work, technical proposals, solution estimates, presentations, and client demonstrations.
• Participate in client meetings as the technical and delivery authority for Databricks‑related work.

Required Qualifications

• Bachelor’s degree in computer science, information technology, data engineering, engineering, or a related discipline.
• At least 10 years of experience in data engineering, data architecture, analytics engineering, or related technology roles.
• At least 5 years of hands‑on experience designing and implementing solutions using Databricks.At least 3 years of experience managing or formally leading technical engineering teams.
• Advanced experience with:
  • Databricks Lakehouse Platform
  • Apache Spark and PySpark
  • Spark SQL and advanced SQL development
  • Delta Lake and medallion architecture
  • Unity Catalog and enterprise data governance
  • ETL and ELT pipeline architecture
  • Batch and real‑time data processing
  • Data modeling and data warehousing
  • Python‑based data engineering
  • Databricks Workflows, Jobs, and cluster management
• Experience deploying Databricks solutions in Azure, AWS, or Google Cloud.
• Experience with CI/CD, Git‑based development, automated testing, and infrastructure as code.
• Demonstrated ability to optimize Spark workloads, cluster configurations, query performance, reliability, and cloud costs.
• Experience managing technical delivery, resource assignments, risks, schedules, and client expectations.
• Strong written, verbal, presentation, documentation, and stakeholder‑management skills.
• Ability to explain complex technical concepts to executives, business stakeholders, and nontechnical audiences.

Preferred Qualifications

• Databricks Certified Data Engineer Professional, Databricks Certified Data Engineer Associate, or Databricks Certified Machine Learning Professional.
• Databricks Certified Data Architect or comparable advanced architecture credentials.
• Microsoft Azure, AWS, or Google Cloud professional‑level certification.
• Experience working in a consulting, professional‑services, systems‑integration, or managed‑services environment.
• Experience supporting federal, state, or local government clients.
• Experience working with major consulting or systems‑integration partners.
• Knowledge of federal security, privacy, governance, and compliance requirements.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Terraform.
• Experience with MLflow, MLOps, generative AI, Databricks Mosaic AI, vector search, or machine‑learning deployment.
• Familiarity with data standards, metadata frameworks, data catalogs, data‑sharing protocols, and open‑data environments.
• Experience managing geographically distributed or remote technical teams.
• Experience contributing to proposals, technical responses, statements of work, and project estimates.

Leadership Competencies

The successful candidate will demonstrate:

• Hands‑on technical credibility and sound architectural judgment.
• The ability to lead without becoming disconnected from the technology.
• Strong accountability for team performance and project outcomes.
• Effective coaching, delegation, and conflict‑resolution skills.
• Clear and proactive communication with clients and internal leadership.
• The ability to manage competing priorities in a fast‑paced consulting environment.
• A commitment to quality, security, documentation, and continuous improvement.

Success Measures

Performance in this role will be evaluated based on:

• Quality, scalability, security, and reliability of Databricks solutions.
• On‑time and within‑budget delivery of client commitments.
• Team performance, retention, development, and technical growth.
• Client satisfaction and effective stakeholder communication.
• Reduction in delivery risks, production incidents, and technical debt.
• Adoption of standardized architectures, engineering practices, and reusable solutions.
• Effective management of Databricks consumption, infrastructure, and cloud costs.
• Growth and maturity of the organization’s Databricks practice.