Databricks Practice Lead / Engineering Manager

Scicom Infrastructure Services, Inc.; United States; On-site; USD 170,000 - 260,000; Full time

Job summary

Scicom Infrastructure Services seeks a senior Databricks Practice Lead to provide hands-on technical leadership while managing a team of data engineers, architects, and consultants supporting enterprise programs. This role leads design and governance of modern data platforms and guides delivery teams across client engagements.

The position requires deep Databricks expertise, cloud experience (Azure, AWS, or GCP), and strong people leadership to drive secure, scalable solutions and timely

Qualifications

• Bachelor's degree or higher in a related field.
• Minimum 10 years in data engineering/architecture roles with 5+ years in Databricks design/implementation.

Responsibilities

• Provide hands-on technical leadership for Databricks projects.
• Lead a team of data engineers, architects, and consultants.
• Define architectural standards and governance frameworks.
• Oversee platform migrations from legacy environments to Databricks Lakehouse.
• Engage with clients to present solutions and ensure delivery excellence.

Skills

Databricks

Leadership

Cloud platforms

Python / PySpark

ETL / ELT

CI/CD

Education

Bachelor's degree in CS/ENG or related

Tools

Databricks Lakehouse

Azure/AWS/Google Cloud

Unity Catalog

MLflow

Job description

Position Summary Scicom Infrastructure Services is seeking an experiencedDatabricks Practice Lead / Engineering Managerto provide hands-on technical leadership while managing a team of data engineers, architects, and consultants supporting complex enterprise and government programs. This role requires a senior Databricks expert who can design and oversee modern data platforms, establish technical standards, guide delivery teams, and remain actively involved in architecture, troubleshooting, code reviews, and client-facing solution development. The successful candidate will balance deep technical expertise with strong people leadership, delivery management, and stakeholder communication skills.

Key Responsibilities

Databricks Technical Leadership

• Serve as the organization’s subject-matter expert for the Databricks Lakehouse Platform.
• Design scalable, secure, and highly available data architectures using Databricks, Apache Spark, Delta Lake, and cloud-native technologies.
• Lead the implementation of batch, streaming, ETL, ELT, analytics, machine-learning, and AI-enabled data solutions.
• Define architectural standards for medallion architectures, data modeling, ingestion, transformation, orchestration, and data consumption.
• Establish governance frameworks using Unity Catalog, including data lineage, access controls, auditing, metadata management, and secure data sharing.
• Guide Databricks workspace design, cluster configuration, serverless computing, workload isolation, performance tuning, and cost optimization.
• Oversee integration between Databricks and cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
• Develop or review solutions involving PySpark, Spark SQL, Python, Delta Live Tables, Structured Streaming, Auto Loader, MLflow, and Databricks Workflows.
• Lead platform migrations and modernization efforts from legacy databases, data warehouses, Hadoop environments, and traditional ETL platforms.
• Establish development standards for source control, automated testing, CI/CD, infrastructure as code, monitoring, and production support.
• Conduct architecture reviews, code reviews, technical assessments, and root-cause analyses.
• Evaluate emerging Databricks capabilities and recommend appropriate adoption strategies.

Team Leadership and Management

• Manage, mentor, and develop a team of Databricks engineers, data engineers, architects, and technical consultants.
• Assign resources and responsibilities based on project needs, employee strengths, availability, and technical complexity.
• Establish measurable goals, performance expectations, development plans, and technical competency standards.
• Conduct regular one-on-one meetings, performance reviews, coaching sessions, and technical development activities.
• Support recruiting, interviewing, candidate evaluation, onboarding, and workforce planning.
• Identify technical or performance gaps and coordinate training, mentoring, or corrective action as appropriate.
• Promote collaboration, accountability, documentation, knowledge sharing, and continuous improvement.
• Develop reusable accelerators, reference architectures, templates, and delivery playbooks.
• Build and maintain a strong Databricks practice capable of supporting multiple concurrent client engagements.

Program and Delivery Management

• Provide delivery oversight for Databricks and data-engineering projects from planning through implementation and operational support.
• Translate business, functional, security, and contractual requirements into technical plans and deliverables.
• Develop project estimates, staffing plans, delivery schedules, milestones, and risk‑mitigation strategies.
• Monitor project scope, schedule, quality, budget, resource utilization, dependencies, and technical risks.
• Ensure deliverables meet client requirements, internal quality standards, security controls, and contractual commitments.
• Coordinate work across engineering, cloud, cybersecurity, data governance, analytics, project‑management, and client teams.
• Track delivery metrics and provide clear status reports to internal leadership, clients, and program stakeholders.
• Lead technical escalations and ensure issues are resolved promptly and appropriately documented.
• Support statements of work, technical proposals, solution estimates, presentations, and client demonstrations.
• Participate in client meetings as the technical and delivery authority for Databricks‑related work.

Required Qualifications

• Bachelor’s degree in computer science, information technology, data engineering, engineering, or a related discipline.
• At least 10 years of experience in data engineering, data architecture, analytics engineering, or related technology roles.
• At least 5 years of hands‑on experience designing and implementing solutions using Databricks.
• At least 3 years of experience managing or formally leading technical engineering teams.
• Advanced experience with:
  • Databricks Lakehouse Platform
  • Apache Spark and PySpark
  • Spark SQL and advanced SQL development
  • Delta Lake and medallion architecture
  • Unity Catalog and enterprise data governance
  • ETL and ELT pipeline architecture
  • Batch and real‑time data processing
  • Data modeling and data warehousing
  • Python‑based data engineering
  • Databricks Workflows, Jobs, and cluster management
  • Experience deploying Databricks solutions in Azure, AWS, or Google Cloud.
  • Experience with CI/CD, Git‑based development, automated testing, and infrastructure as code.
  • Demonstrated ability to optimize Spark workloads, cluster configurations, query performance, reliability, and cloud costs.
  • Experience managing technical delivery, resource assignments, risks, schedules, and client expectations.
  • Strong written, verbal, presentation, documentation, and stakeholder‑management skills.
  • Ability to explain complex technical concepts to executives, business stakeholders, and nontechnical audiences.

Preferred Qualifications

• Databricks Certified Data Engineer Professional, Databricks Certified Data Engineer Associate, or Databricks Certified Machine Learning Professional.
• Databricks Certified Data Architect or comparable advanced architecture credentials.
• Microsoft Azure, AWS, or Google Cloud professional‑level certification.
• Experience working in a consulting, professional‑services, systems‑integration, or managed‑services environment.
• Experience supporting federal, state, or local government clients.
• Experience working with major consulting or systems‑integration partners.
• Knowledge of federal security, privacy, governance, and compliance requirements.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Terraform.
• Experience with MLflow, MLOps, generative AI, Databricks Mosaic AI, vector search, or machine‑learning deployment.
• Familiarity with data standards, metadata frameworks, data catalogs, data‑sharing protocols, and open‑data environments.
• Experience managing geographically distributed or remote technical teams.
• Experience contributing to proposals, technical responses, statements of work, and project estimates.

Leadership Competencies

• Hands‑on technical credibility and sound architectural judgment.
• The ability to lead without becoming disconnected from the technology.
• Strong accountability for team performance and project outcomes.
• Effective coaching, delegation, and conflict‑resolution skills.
• Clear and proactive communication with clients and internal leadership.
• The ability to manage competing priorities in a fast‑paced consulting environment.
• A commitment to quality, security, documentation, and continuous improvement.

Success Measures Performance in this role will be evaluated based on:

• Quality, scalability, security, and reliability of Databricks solutions.
• On‑time and within‑budget delivery of client commitments.
• Team performance, retention, development, and technical growth.
• Client satisfaction and effective stakeholder communication.
• Reduction in delivery risks, production incidents, and technical debt.
• Adoption of standardized architectures, engineering practices, and reusable solutions.
• Effective management of Databricks consumption, infrastructure, and cloud costs.
• Growth and maturity of the organization’s Databricks practice.

Originally posted on Himalayas