Data Platform Engineer

INTELLISWIFT INC; San Jose (CA); On-site; USD 150,000 - 200,000; Full time

Job summary

An innovative company is seeking a Data Platform Engineer to enhance their cloud infrastructure. In this role, you will manage and optimize production-scale environments on platforms like AWS and Azure, ensuring efficient data storage and compliance with security standards. You will develop automation tools, monitor cloud spending, and explore integration of GenAI technologies to improve user experience. This position offers a dynamic work environment where your expertise will directly impact the effectiveness of data operations and platform enhancements. If you are passionate about cloud technologies and data management, this opportunity is perfect for you.

Qualifications

• Experience in cloud infrastructure administration with AWS and Azure.
• Proficient in data storage technologies like ADLS Gen2 and AWS S3.

Responsibilities

• Setup and maintain production scale Databricks environment on AWS and Azure.
• Develop tools for automation and configuration management to support daily operations.

Skills

Cloud Infrastructure Administration and Automation

AWS

Azure

ADLS Gen2

AWS S3

Hive or MySQL

MongoDB

Vector Databases

Databricks Workspace

Airflow

Kubernetes

Jira

Service Now

Education

BS in Computer Science

BS in Computer Engineering

Tools

Databricks

Prometheus

Splunk

ELK

PowerBI

Job description

Job Title: Data Platform Engineer

Location: San Jose, CA 95110

Duration: 4 months

Contract Type: W2 only

Pay Rate: $79.58/hour

Duties:

1. Setup and maintain production scale Databricks environment on public cloud such as Microsoft Azure and AWS (Amazon Web Services)
2. Setup and maintain production scale data storage such as ADLS (Azure Data Lake Storage) and AWS S3 for multiple tenant teams using our Data Platform
3. Setup and maintain production scale micro services to support the daily operation of our data platform. Services include job scheduling, security, financial, and administrative services, etc.
4. Provide triage and guidance to the team on various support issues raised by our tenants
5. Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting to assist with daily RTB (Running the Business) operations
6. Budget and monitor cloud spend, always think of ways to avoid cloud resource wastage, utilize 3rd party tools, or develop your own tools to help the team with cost optimization
7. Assure security and privacy compliance and implement Adobe Security & Compliance solutions to lock down data stored in our data lake
8. Explore GenAI technologies and find opportunity to integrate them with our data platform, providing platform enhancement or improving platform user experience in the end
9. Work with various 3rd party vendors for troubleshooting, proof of concept, and other collaborative projects to enhance our product.

Skills:

1. Cloud Infrastructure Administration and Automation: AWS, Azure
2. Proficient with following storage technologies: ADLS Gen2, AWS S3, Hive or MySQL, MongoDB, Vector Databases
3. Setup, troubleshoot and maintain following technologies: Databricks Workspace, includes but not limited to - Unity Catalog, Vector Search, SQL Warehouse, Serverless Compute, Spark workloads, Airflow and DAGs, Azure Kubernetes Service or Elastic Kubernetes Service, Collibra, Neo4J, Metric Insights
4. Ability to setup monitoring and alerting with: Databricks System Tables, Prometheus, Splunk, ELK, PowerBI
5. Familiar with how to troubleshoot and maintain: Servers with Linux system, Kubernetes environment
6. Knowledge to operate with: Jira, Service Now

Education: BS in Computer Science, Computer Engineering, or similar