AI Data Engineer — Scale Pipelines (Hybrid)

AVP VIGILANT TECHNOLOGY PVT LTD; San Francisco (CA); On-site; USD 115,000 - 195,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

401(k) retirement benefits

Paid time off and holidays

Professional development opportunities

Flexible/hybrid work options

Job summary

AVP Vigilant Technology Pvt Ltd is seeking an AI Data Engineer to design and build scalable data pipelines, cloud-based data platforms, and high-quality datasets for modern AI applications. Strong Python and SQL skills and hands-on cloud experience are required.

You will work with Data Scientists, ML Engineers, Software Engineers, and business teams to deliver reliable data infrastructure, ETL processes, data warehouses, and analytics-ready datasets across multiple locations including San

Qualifications

• 2–7 years of experience in Data Engineering, AI/ML Engineering, or a related field.
• Strong hands-on experience with Python and SQL.
• Solid understanding of data engineering, data modeling, ETL/ELT, and data warehousing.
• Working knowledge of machine learning concepts and AI data workflows.
• Experience with at least one major cloud platform: AWS, Azure, or Google Cloud.
• Experience working with relational and analytical databases.
• Strong understanding of data integration, APIs, and distributed data processing.
• Ability to work effectively in a collaborative, fast-paced environment.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT and data pipelines.
• Build and optimize data infrastructure supporting AI/ML applications.
• Develop reliable data processing solutions using Python and SQL.
• Integrate data from databases, APIs, applications, and other sources.
• Design data models, data warehouses, and analytics-ready datasets.
• Prepare, transform, validate, and optimize datasets for machine learning.
• Collaborate with Data Scientists, ML Engineers, Software Engineers, and business teams.
• Implement data quality, monitoring, governance, and security best practices.
• Optimize data pipelines and cloud infrastructure for performance, scalability, and reliability.
• Support ML model training, deployment, and production data workflows.
• Troubleshoot data and pipeline issues and continuously improve system performance.

Skills

Python

SQL

Data modeling

ETL/ELT

Data warehousing

APIs

Distributed data processing

Tools

AWS

Azure

Google Cloud

Apache Spark

Databricks

Airflow

Kafka

Snowflake

BigQuery

Redshift

Docker

Kubernetes

CI/CD

Job description

AVP Vigilant Technology Pvt Ltd is seeking an AI Data Engineer to design and build scalable data pipelines, cloud-based data platforms, and high-quality datasets for modern AI applications. Strong Python and SQL skills and hands-on cloud experience are required.

You will work with Data Scientists, ML Engineers, Software Engineers, and business teams to deliver reliable data infrastructure, ETL processes, data warehouses, and analytics-ready datasets across multiple locations including San