Data Engineer – Architect

NStarX Inc.; Hyderabad; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

NStarX Inc. seeks an Data Engineer – Architect to design scalable, secure data lakehouses and warehouses using Medallion patterns. Lead engineering standards, review architectures, and mentor engineers across PySpark, SQL, and Python workloads.

You will architect end-to-end Azure data platforms, optimize Delta Lake storage, and drive cost efficiency. Collaboration with analytics, ML, and BI teams is expected to deliver high-performance data solutions.

Qualifications

• 10+ years in IT/software engineering with at least 5 years in Azure data architecture or equivalent data-platform design.
• 10+ years with distributed data processing or cloud data platforms, with Databricks, Spark, Snowflake exposure.
• Strong understanding of Lakehouse/Warehouse architecture, Medallion Architecture, scalability, security, and cost optimization.
• Mastery of Kimball dimensional modelling, Star/Snowflake schemas, Data Vault, and analytical data design.
• Proficient in Python/PySpark and SQL, with production-grade transformations and performance tuning.
• Experience with Airflow, Azure Data Factory, Databricks Workflows, or similar orchestration tools.
• Git, CI/CD, Infrastructure-as-Code (Terraform/Bicep), and automated deployment practices.
• Deep knowledge of Azure services, RBAC, identity, network isolation, security controls, and governance.
• Strong communication and leadership to mentor engineers and collaborate with stakeholders.

Skills

Data architecture

Medallion Architecture

Azure Cloud

Python/PySpark

SQL

CI/CD

Leadership

Communication

Tools

Databricks

Spark

Snowflake

Airflow

Azure Data Factory

Terraform

Unity Catalog

Purview

Delta Lake

Job description

Data Engineer – Architect

KEY RESPONSIBILITIES

• System Architecture: Design scalable, resilient, secure, and cost-effective data lakehouses and warehouses using modern patterns such as Medallion Architecture (Bronze/Silver/Gold).
• Technical Leadership: Set engineering standards, conduct architecture and code reviews, establish reusable frameworks, and mentor data engineers on best practices.
• Data Enablement: Deliver reliable, high-performance data platforms that support BI, data science, machine learning, and operational analytics.
• Cloud Architecture: Design end-to-end Azure/cloud infrastructure blueprints covering storage, compute, RBAC, security, network isolation, scalability, and cost optimization.
• Big Data & Processing: Architect and optimize Spark/Databricks/Snowflake workloads, including Delta Lake design, cluster sizing, partitioning, Z-Ordering, Vacuuming, clustering, and performance tuning.
• Data Pipelines & Ingestion: Build and standardize production-grade PySpark, Python, and SQL pipelines supporting batch, micro-batch, streaming, and CDC ingestion patterns.
• Data Modeling: Establish organizational data-modeling standards and design Star/Snowflake schemas, Kimball dimensional models, Data Vault patterns, and Lakehouse storage layers.
• Orchestration: Architect workflow dependencies, retries, alerts, monitoring, and backfilling using Azure Data Factory, Airflow, or Databricks Workflows.
• DevOps & CI/CD: Implement Git-based development standards and automated CI/CD pipelines using Azure DevOps or GitHub Actions for reliable data-platform deployments.
• Infrastructure-as-Code: Design and implement infrastructure provisioning using Terraform, Bicep, or equivalent IaC frameworks.
• Governance & Security: Implement metadata catalogs, lineage, data quality controls, access policies, and regulatory compliance practices using technologies such as Unity Catalog and Microsoft Purview.
• Performance & FinOps: Benchmark workloads, resolve bottlenecks, optimize memory and compute configurations, monitor cloud consumption, and drive cost-efficiency across data platforms.

MINIMUM REQUIREMENTS

• Overall Experience: 10 to 12+ years of IT/software engineering experience, with at least 5 years in Azure data architecture, cloud platform design, or equivalent data-platform architecture.
• Distributed Computing: 10+ years of experience working with distributed data processing or modern cloud data platforms, with strong hands-on exposure to Databricks, Spark, Snowflake, or equivalent technologies.
• Data Architecture: Strong understanding of Lakehouse/Warehouse architecture, Medallion Architecture, scalability, resiliency, security, and cost optimization.
• Data Modelling: Strong mastery of Kimball dimensional modelling, Star/Snowflake schemas, Data Vault, and analytical data design.
• Programming: Strong hands-on expertise in Python/PySpark and SQL, including production-grade transformations and performance optimization.
• Orchestration: Strong experience with Airflow, Azure Data Factory, Databricks Workflows, or equivalent orchestration platforms.
• DevOps: Hands-on experience with Git, CI/CD, Infrastructure-as-Code, Terraform/Bicep, and automated deployment practices.
• Cloud & Security: Strong understanding of Azure services, RBAC, identity, network isolation, security controls, monitoring, and enterprise governance.
• Communication & Leadership: Ability to communicate architecture decisions clearly, lead technical discussions, mentor engineers, and work effectively with cross-functional stakeholders.

GOOD TO HAVE

• Governance: Experience with Unity Catalog, Microsoft Purview, metadata management, data lineage, data quality, GDPR, HIPAA, or other regulated-data environments.
• Monitoring: Experience with Azure Monitor, Log Analytics, operational alerting, and platform observability.
• Cloud Breadth: Exposure to AWS or GCP data platforms in addition to Azure.
• Modern Data Engineering: Experience with CDC, streaming architectures, reusable ingestion frameworks, and platform engineering practices.