Data Architect

Soul Ai; Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Soul Ai in Bengaluru is seeking a senior data platform engineer to design scalable data platforms, integrate diverse data sources, and build real-time and batch pipelines for analytics and ML. You will implement streaming solutions and drive DevOps/DataOps practices across cloud environments.

Candidates should demonstrate expertise in distributed data systems, programming languages (Python/Scala/Java), and experience with Spark, Kafka, and major cloud platforms.

Qualifications

• Experience designing scalable, distributed data systems.
• Proficiency in Python/Scala/Java for data engineering tasks.
• Hands-on with Spark, Hadoop, Flink, Kafka and cloud platforms.

Responsibilities

• Design and architect enterprise-scale data platforms, integrating diverse data sources and tools
• Develop real-time and batch data pipelines to support analytics and machine learning
• Define and enforce data governance strategies for security, integrity and compliance; optimize pipelines for cloud environments
• Implement real-time streaming solutions (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices

Skills

Distributed data systems

Python/Scala/Java

Streaming data pipelines

Cloud platforms

CI/CD & DevOps tooling

Tools

Terraform

CloudFormation

Airflow

Kubernetes

Prometheus

Grafana

Job description

Responsibilities:

• Design and architect enterprise-scale data platforms, integrating diverse data sources and tools


• Develop real-time and batch data pipelines to support analytics and machine learning


• Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments


• Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices

Required Skills:

• Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP)


• Proficient in data modeling, governance, warehousing (Snowflake, Redshift, Big Query), and security/compliance standards (GDPR, HIPAA)


• Hands-on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana)

Nice to Have:

• Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions


• Contributions to open-source data engineering communities