Data Architect

Soul Ai; Bengaluru; Sur place; INR 2 000 000 - 4 200 000; Plein temps

Résumé du poste

Soul Ai is seeking a data engineering expert to design and build enterprise-scale data platforms that ingest diverse sources and support analytics and ML workloads.

You will develop real-time and batch pipelines, establish data governance, and optimize pipelines for performance, cost, and security in cloud environments. Experience with streaming tech and DevOps/DataOps practices is essential.

Qualifications

• Design scalable, distributed data systems and programming in Python/Scala/Java.
• Expertise in Spark, Hadoop, Flink, Kafka and cloud platforms (AWS, Azure, GCP).
• Hands-on CI/CD with Terraform, CloudFormation, Airflow, Kubernetes; optimize data infra with Prometheus and Grafana.

Responsabilités

• Design and architect enterprise-scale data platforms integrating diverse sources.
• Develop real-time and batch data pipelines to support analytics and ML.
• Define and enforce data governance for security, integrity, and compliance in cloud environments.
• Implement real-time streaming solutions (Kafka, Kinesis, Flink) and DevOps/DataOps practices.

Connaissances

Distributed data systems

Python/Scala/Java

Apache Spark

Hadoop

Flink

Kafka

Cloud platforms

Outils

Terraform

CloudFormation

Airflow

Kubernetes

Prometheus

Grafana

Description du poste

Responsibilities:

• Design and architect enterprise-scale data platforms, integrating diverse data sources and tools
• Develop real-time and batch data pipelines to support analytics and machine learning
• Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments
• Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices

Required Skills:

• Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP)
• Proficient in data modeling, governance, warehousing (Snowflake, Redshift, Big Query), and security/compliance standards (GDPR, HIPAA)
• Hands‑on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana)

Nice to Have:

• Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions
• Contributions to open‑source data engineering communities