Data Architect

Soul Ai; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Graph databases experience

Open-source contributions

Job summary

Soul Ai is seeking a seasoned Data Platform Architect in Hyderabad to design and implement enterprise-scale data platforms, unifying diverse sources and tools. You will build real-time and batch pipelines for analytics and ML, while enforcing governance to secure data and optimize cloud performance.

Experience with Spark, Hadoop, Flink, Kafka and cloud ecosystems (AWS/Azure/GCP) is essential, along with CI/CD and monitoring tooling like Terraform, Airflow, Kubernetes, Prometheus, and Grafana.

Qualifications

• Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP).
• Proficient in data modeling, governance, warehousing (Snowflake, Redshift, Big Query), and security/compliance standards (GDPR, HIPAA).
• Hands-on experience with CI/CD (Terraform, CloudFormation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana).

Responsibilities

• Design and architect enterprise-scale data platforms, integrating diverse data sources and tools
• Develop real-time and batch data pipelines to support analytics and machine learning
• Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments
• Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices

Skills

Python

Scala

Java

Apache Spark

Hadoop

Flink

Kafka

Cloud platforms

AWS

Azure

GCP

Data governance

Data modeling

CI/CD

Kubernetes

Prometheus

Grafana

Tools

Terraform

CloudFormation

Airflow

Kubernetes

Prometheus

Grafana

Job description

Responsibilities

• Design and architect enterprise-scale data platforms, integrating diverse data sources and tools
• Develop real-time and batch data pipelines to support analytics and machine learning
• Define and enforce data governance strategies to ensure security, integrity, and compliance along with optimizing data pipelines for high performance, scalability, and cost efficiency in cloud environments
• Implement solutions for real-time streaming data (Kafka, AWS Kinesis, Apache Flink) and adopt DevOps/DataOps best practices

Required Skills

• Strong experience in designing scalable, distributed data systems and programming (Python, Scala, Java) with expertise in Apache Spark, Hadoop, Flink, Kafka, and cloud platforms (AWS, Azure, GCP)
• Proficient in data modeling, governance, warehousing (Snowflake, Redshift, Big Query), and security/compliance standards (GDPR, HIPAA)
• Hands-on experience with CI/CD (Terraform, Cloud Formation, Airflow, Kubernetes) and data infrastructure optimization (Prometheus, Grafana)

Nice to Have

• Experience with graph databases, machine learning pipeline integration, real-time analytics, and IoT solutions
• Contributions to open-source data engineering communities