Senior AI Data Engineer: Databricks, Spark & ML Pipelines

HCL Global Systems Inc; Princeton (NJ); On-site; USD 150,000 - 190,000; Full time

Job summary

Visa seeks a Senior AI Data Engineer to design and operate AI/ML data pipelines, model metadata, and data lineage. You will build scalable infrastructure with Databricks and Spark, implement MCP servers for AI data distribution, and apply MLflow for experiment tracking and registry.

Collaboration with data scientists and ML engineers is essential. The ideal candidate has 10+ years in data roles and 5+ years in data engineering, plus experience with ML workloads and governance.

Qualifications

• 10+ years of overall experience in data roles.
• 5+ years hands-on data engineering.
• 2+ years with AI/ML workloads.
• Experience with Databricks and Spark.
• Understanding ML lifecycle: training, deployment, monitoring.

Responsibilities

• Design and build AI/ML data pipelines for model metadata, data lineage, and performance tracking.
• Build scalable data infrastructure using Databricks & Spark.
• Develop MCP servers and enable AI data distribution through MCP.
• Build feature engineering and data preprocessing pipelines.
• Implement MLflow for experiment tracking, model versioning, and model registry.
• Develop AI agent discovery, classification, and inventory workflows.
• Design knowledge graphs for AI model relationships, dependencies, and data lineage.
• Build real-time pipelines for model monitoring, drift detection, and performance tracking.
• Implement AI-specific data quality, governance, security, and compliance frameworks.
• Collaborate closely with Data Scientists, ML Engineers, and Architects

Skills

Data engineering

AI/ML pipelines

ML lifecycle management

Model governance

Real-time data pipelines

Collaboration

Tools

Databricks

Apache Spark

MLflow

MCP

Vector databases

Embeddings

Job description

Visa seeks a Senior AI Data Engineer to design and operate AI/ML data pipelines, model metadata, and data lineage. You will build scalable infrastructure with Databricks and Spark, implement MCP servers for AI data distribution, and apply MLflow for experiment tracking and registry.

Collaboration with data scientists and ML engineers is essential. The ideal candidate has 10+ years in data roles and 5+ years in data engineering, plus experience with ML workloads and governance.