Manager - Data Lake and Data Architecture - AWS, Azure, Good to have GCP

Sirius AI; Gurugram District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

A technology company is seeking a Manager for Data Lake and Data Architecture. This role involves designing scalable architectures and developing data pipelines using AWS and Azure, among other responsibilities. The ideal candidate should have over 6 years of experience in data engineering and expertise in various cloud platforms and big data technologies. Competitive compensation and growth opportunities are offered.

Qualifications

• 6+ years of experience as a Data Engineer, specializing in cloud-agnostic data solutions.
• Strong expertise in cloud data platforms with hands-on experience.
• Proficiency in big data technologies such as Apache Spark, Hadoop, and Kafka.

Responsibilities

• Design and implement scalable, cloud-agnostic Data Lake architectures.
• Develop and maintain robust data pipelines for analytics.
• Implement data governance policies and security frameworks.

Skills

Data Engineering

Cloud Data Solutions

Big Data Technologies

Problem-Solving

Containerization

Education

Bachelor's degree in Computer Science

Tools

AWS

Azure

Google Cloud

Apache Spark

Kubernetes

Docker

Job description

Manager - Data Lake and Data Architecture - AWS, Azure, Good to have GCP

Get AI-powered advice on this job and more exclusive features.

Direct message the job poster from Sirius AI

Key Responsibilities

• Data Architecture & Management: Design and implement scalable, cloud-agnostic Data Lake, Data LakeHouse, Data Mesh and Data Fabric architectures to efficiently store, process, and manage structured and unstructured data from various sources.
• Data Pipeline Development: Design, develop, and maintain robust data pipelines to ingest, process, and transform data from multiple sources into usable formats for analytics and reporting using services like AWS Glue, Azure Data Factory, GCP Dataflow, Apache Spark, or Apache Airflow.
• Data Integration and ETL: Develop and optimize Extract, Transform, Load (ETL) and ELT processes to integrate disparate data sources into the data lake, ensuring high data quality, consistency, and reliability across multiple cloud platforms.
• Cloud-Agnostic Data Engineering: Develop data solutions that are cloud-agnostic, leveraging open-source technologies like Apache Spark, Delta Lake, Presto, and Kubernetes, ensuring compatibility across AWS, Azure, and GCP.
• Big Data Processing & Analytics: Utilize big data technologies such as Apache Spark, Hive, and Presto for distributed computing, enabling large-scale data transformations and analytics.
• Data Governance and Security: Implement robust data governance policies, security frameworks, and compliance controls, including role-based access control (RBAC), encryption, and monitoring to meet industry standards (GDPR, HIPAA, PCI-DSS).
• DevOps Integration for Data Platforms: Leverage cloud-agnostic DevOps tools and practices for source control, build automation, release management, and Infrastructure as Code (IaC) to streamline the development, deployment, and management of data lake and data architecture solutions across multiple cloud providers. Solutions should support CI/CD pipelines, automated testing, and scalable data workflows.
• Continuous Integration and Deployment (CI/CD): Establish automated CI/CD pipelines to streamline deployment, testing, and monitoring of data infrastructure and workflows.
• Performance Optimization: Optimize data workflows and query performance using indexing, caching, and partitioning strategies to improve efficiency and cost-effectiveness.
• Monitoring and Troubleshooting: Implement observability solutions using tools like Prometheus, Grafana, or cloud-native monitoring services to proactively detect and resolve data pipeline issues.
• Collaboration and Documentation: Work with cross-functional teams, including data scientists, analysts, and business stakeholders, to design and implement scalable data solutions. Maintain comprehensive documentation of data architectures, processes, and best practices.

Job Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 6+ years of experience as a Data Engineer, specializing in cloud-agnostic data solutions and data lake architectures.
• Strong expertise in cloud data platforms such as AWS, Azure, and Google Cloud, with hands-on experience in services like AWS S3, Azure Data Lake, Google Cloud Storage, and related data processing tools.
• Proficiency in big data technologies such as Apache Spark, Hadoop, Kafka, Delta Lake, or Presto.
• Experience with SQL and NoSQL databases, including PostgreSQL, MySQL, and DynamoDB.
• Expertise in containerization and orchestration platforms such as Docker and Kubernetes.
• Experience implementing DevOps and CI/CD practices using Terraform, CloudFormation, or other Infrastructure as Code (IaC) tools.
• Knowledge of data visualization tools such as Power BI, Tableau, or Looker for presenting insights and reports.
• Strong problem-solving and troubleshooting skills with a proactive approach to identifying and resolving issues.
• Experience leading teams of 5+ cloud engineers.
• Preferred certifications in AWS, Azure, or Google Cloud data engineering.

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• IT Services and IT Consulting