Cloud and Data Platform Engineer

Indigo Beam; Houston (TX); Remote; USD 120,000 - 180,000; Full time

Job summary

Indigo Beam Consulting in the United States is seeking a Cloud and Data Platform Engineer to design, implement, and optimize our Azure cloud and analytics platform, including Microsoft Fabric and Azure Databricks. This role blends infrastructure engineering with data platform administration to deliver secure, scalable services for analytics, forecasting, and machine learning workloads.

You will collaborate with Data Engineering, Data Science, and IT teams, automate provisioning with IaC, and

Qualifications

• Bachelor's degree or equivalent experience in CS/IT/Engineering.
• 3+ years administering cloud data platforms with Azure Fabric and Azure Databricks.
• Solid Azure engineering across networking, IAM, compute, storage, DR.
• Hands-on Databricks administration including workspaces, clusters, jobs, SQL warehouses.
• Experience with IaC and CI/CD practices.

Responsibilities

• Administer Microsoft Fabric and Azure Databricks environments.
• Configure Databricks workspaces, clusters, SQL warehouses, jobs, and Unity Catalog.
• Design and manage Azure networking, identity, compute, storage, and serverless services.
• Develop IaC modules with Terraform/Ansible/ARM templates and CI/CD pipelines.
• Implement security, monitoring, and runbooks for platform reliability.

Skills

Azure engineering

Databricks administration

CI/CD pipelines

Python scripting

Networking and IAM

Governance and security

Education

Bachelor's degree in CS/IT/Engineering

Tools

Terraform

Ansible

ARM templates

Azure DevOps

GitHub Actions

Unity Catalog

Databricks

Job description

Indigo Beam Consulting provides a common-sense approach to solving business problems through technology.

Born out of Bluware Inc. which has been in business for twenty-seven years, Indigo Beam was conceived as a full-service business consulting company that deeply cares about creating technology solutions that are in harmony with the environments in which they exist.

We believe that for a technology solution to be sustainable, it must be congruent with the business processes it serves, indigenously maintainable, and cognizant of the unique traits of the organization which it calls home.

Job Description

This is a remote position.

Role: Cloud and Data Platform Engineer

Location: Remote (U.S.)

Contract position: 6+months

Position Summary

The Cloud and Data Platform Engineer designs, implements, administers, and optimizes the company’s Azure cloud and enterprise analytics platforms, including Microsoft Fabric and Azure Databricks. The role combines day-to-day cloud infrastructure engineering with data platform administration to provide secure, reliable, scalable, and cost-effective services for analytics, forecasting, and machine learning workloads.

Working with Data Engineering, Data Science, Forecasting, Enterprise Architecture, Cybersecurity, IT Operations, and business teams, this engineer supports platform connectivity, automated provisioning and deployment, governance, and operational resilience across the data and analytics environment.

Key Responsibilities

Cloud Infrastructure and Data Platform Administration

• Administer Microsoft Fabric and Azure Databricks environments, including platform configuration, access, capacity, monitoring, and lifecycle management.
  
• Configure and manage Databricks workspaces, clusters, SQL warehouses, jobs, workflows, Unity Catalog, permissions, and service principals.
  
• Design, provision, manage, and troubleshoot Azure networking, identity, compute, storage, and serverless services supporting enterprise analytics.
  
• Build and maintain secure, scalable environments for ingestion, transformation, feature engineering, model training, and forecast generation, including demand, supply, sales, and operational forecasting.
  
• Participate in the design and implementation of connectivity between Azure platforms and SAP, SAP Datasphere, SQL platforms, Snowflake, and external data providers.
  

Automation and Deployment

• Develop, maintain, and version reusable Infrastructure as Code modules and templates using Terraform, Ansible, ARM templates, or equivalent tools.
  
• Implement GitOps and declarative infrastructure deployment practices aligned with enterprise cloud standards.
  
• Establish and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools to deploy infrastructure, Databricks notebooks, code, workflows, and data products.
  
• Use Python, SQL, and scripting to automate platform operations and improve consistency and supportability.
  

Security Governance and Operational Reliability

• Implement identity and access management, network security, data access governance, secrets management, auditing, and logging controls.
  
• Monitor platform health, pipeline reliability, job performance, capacity, usage, and cost, and optimize compute, storage, cluster configurations, and workload scheduling.
  
• Troubleshoot platform issues, support incident response, perform root-cause analysis, and implement corrective actions.
  
• Support architecture, availability, capacity, performance, resilience, disaster recovery, business continuity, and lifecycle planning.
  
• Develop technical standards, operational documentation, and runbooks. Participate in technical reviews and recommend improvements to governance, performance, cost, and developer experience.
  

Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent relevant experience.
  
• At least 3 years of experience administering or engineering cloud data platforms, including hands-on Microsoft Fabric and Azure Databricks administration.
  
• Solid Azure engineering experience across networking, identity and access management, compute, storage, monitoring, and disaster recovery.
  
• Hands-on Databricks administration experience with workspaces, clusters, jobs, workflows, SQL warehouses, and access controls, plus working knowledge of Unity Catalog.
  
• Experience implementing connectivity between Azure and SAP, and understanding of SAP architecture, including SAP Datasphere, and SQL architecture.
  
• Experience with Infrastructure as Code using Terraform, Ansible, ARM templates, or equivalent tools, and proficiency in the associated tools and languages.
  
• Experience with CI/CD, Git-based source control, and automated deployment practices. Strong Python, SQL, and scripting skills for platform automation.
  
• Understanding of batch and streaming pipelines, data lakes, Delta Lake, and data quality controls.
  
• Ability to independently deliver moderately complex technical work, troubleshoot issues, perform root-cause analysis, and assess security, availability, performance, cost, and supportability.
  
• Effective communication, documentation, collaboration, and stakeholder-management skills across technical and business teams.
  

Preferred Qualifications

• At least 5 years of enterprise Cloud Engineering or Platform Engineering experience, including hybrid environments integrating on-premises systems with Azure.
  
• Experience with advanced Unity Catalog capabilities, MLflow, Delta Live Tables, Databricks Asset Bundles, and model serving.
  
• Experience supporting forecasting, machine learning, supply chain, demand planning, or consumer products analytics workloads.
  
• Snowflake administration and connectivity to external data providers such as Circana and Nielsen, and familiarity with ERP, sales, inventory, promotion, or external market data integrations.
  
• Experience with FinOps, cloud cost optimization, enterprise cloud governance, and platform standardization.
  
• Relevant Azure or Databricks credentials, such as Azure Administrator Associate, Azure DevOps Engineer Expert, Databricks Data Engineer Associate or Professional, or platform administrator accreditation.
  

Success Measures

• Reliable, secure, resilient platform services and controlled, auditable access to sensitive data.
  
• Fewer pipeline failures, faster incident resolution, improved workload performance, and faster deployments.
  
• Optimized Azure and Databricks costs, supported by effective capacity and usage management.
  
• Consistent adoption of automation, engineering standards, and high-quality operational documentation.
  

Core Competencies

Create an inclusive environment, collaborate to deliver, drive accountability, demonstrate agility, improve and innovate.