Senior Data Engineer

Nimmee Rahul Dubey; United States; Remote; USD 140,000 - 190,000; Part time

Job summary

Nimmee Rahul Dubey is seeking a Senior Data Engineer to design, build, and scale a production-grade lakehouse platform on AWS, enabling enterprise analytics, financial reporting, and AI-driven applications. The role requires hands-on expertise with Iceberg, S3, Athena, Redshift Serverless, dbt Cloud, and CDC pipelines from SQL Server, with focus on accuracy, reliability, and cost optimization.

This contract-based, fully remote opportunity aligns with US-working hours and supports multi-tenant

Qualifications

• 10+ years building production-grade data platforms.
• 3+ years with lakehouse architectures using Iceberg, Delta, or Hudi.
• Hands-on with AWS DMS and CDC pipelines from SQL Server.
• Advanced dbt Cloud skills: semantic layer, dbt Mesh, MetricFlow, snapshots, SCD2, testing, and incremental models.
• Experience with S3, Athena, Redshift Serverless, and Lake Formation.
• Strong SQL Server / T-SQL and data warehousing modeling.

Responsibilities

• Design and scale a Medallion Lakehouse using Iceberg, S3, Athena, Redshift Serverless, dbt Cloud, and Lake Formation.
• Build and maintain CDC pipelines from SQL Server databases.
• Develop reliable SQL transformations for billing, invoicing, time tracking, and reporting.
• Implement SCD Type 2, snapshots, incremental processing, and historical data tracking.
• Create multi-tenant data architectures with proper isolation and partitioning.
• Manage dbt Cloud workflows, semantic layers, dbt Mesh, MetricFlow, testing.
• Build governed data models and metrics for analytics and AI agents.
• Establish data quality checks, reconciliation, freshness SLAs, observability, and lineage.
• Optimize query performance, storage costs, Iceberg compaction.
• Manage Iceberg operations: snapshot expiry, time travel, partition evolution.
• Collaborate with app/analytics/AI teams to translate requirements into data products.
• Troubleshoot production issues ensuring data accuracy and availability.

Skills

Lakehouse architecture

Apache Iceberg

AWS DMS

dbt Cloud

SQL Server CDC

SQL / T-SQL

Data modeling

Performance tuning

Cost optimization

Multi-tenant SaaS

Tools

Amazon S3

Athena

Redshift Serverless

AWS Lake Formation

dbt Mesh

MetricFlow

Iceberg tooling

Job description

Job Title: Senior Data Engineer Lakehouse Platform

Experience: 8+ years
Employment Type: Contractual
Work Mode: 100% Remote
Working Hours: 6:00 PM – 3:00 AM IST (US Shift)

About the Role

We are looking for an experienced Senior Data Engineer to design, build, and scale a production-grade data lakehouse platform on AWS. The role involves building a reliable, governed, and scalable data foundation to support enterprise analytics, financial reporting, and AI-driven applications.

The ideal candidate will have strong hands‑on expertise in AWS data services, Apache Iceberg, dbt, SQL Server CDC pipelines, and modern data architecture, with a focus on data accuracy, reliability, performance, and cost optimization.

Key Responsibilities

• Design and scale a Medallion Lakehouse Architecture (Bronze, Silver, and Gold) using Apache Iceberg, Amazon S3, Athena, Redshift Serverless, dbt Cloud, and AWS Lake Formation.
• Build and maintain AWS DMS-based Change Data Capture (CDC) pipelines from SQL Server databases.
• Develop reliable SQL transformations for complex business processes, including billing, invoicing, time tracking, and financial reporting.
• Implement SCD Type 2, snapshots, incremental processing, and historical data tracking.
• Design scalable multi-tenant data architectures with appropriate tenant isolation, partitioning, and skew management.
• Manage dbt Cloud workflows, including semantic layers, dbt Mesh, MetricFlow, testing, and incremental models.
• Build governed data models and metrics to support analytics and AI-agent consumption, including MCP integrations where applicable.
• Establish data quality checks, reconciliation processes, freshness SLAs, observability, and data lineage.
• Optimize query performance, storage costs, Iceberg compaction, and data processing efficiency.
• Manage Iceberg operations, including snapshot expiry, time travel, and partition evolution.
• Collaborate with application, analytics, and AI teams to translate business requirements into reliable data products.
• Troubleshoot production issues and ensure data accuracy, availability, and consistency.

Mandatory Skills and Experience

• 10+ years of experience building and managing production‑grade data platforms.
• At least 3 years of experience with lakehouse architectures using Apache Iceberg, Delta Lake, or Apache Hudi.
• Strong hands‑on experience with AWS DMS and CDC pipelines from SQL Server databases.
• Advanced dbt Cloud skills, including semantic layer, dbt Mesh, MetricFlow, snapshots, SCD2, testing, and incremental models.
• Production experience with Amazon S3, Athena, Redshift Serverless, and AWS Lake Formation.
• Strong SQL Server and T‑SQL skills, including the ability to understand complex legacy application logic.
• Expertise in dimensional modeling, data warehousing, and Kimball/Inmon modeling approaches.
• Experience with multi‑tenant SaaS data platforms at scale.
• Hands‑on knowledge of Iceberg compaction, snapshot management, partition evolution, and time travel.
• Proven experience with financial data reconciliation and business‑critical data accuracy.
• Strong analytical, troubleshooting, and performance‑tuning skills.

Good to Have

• Experience exposing governed metrics or dbt semantic layers to AI agents through MCP.
• Familiarity with AWS Glue, Step Functions, EventBridge, MWAA, and EMR Serverless.
• Experience with zero‑ETL integrations, such as Aurora to Redshift or OpenSearch.
• Experience in fintech, legal technology, or other regulated industries.
• Knowledge of query‑plan analysis, Redshift sort keys, and Iceberg file‑size optimization.
• Exposure to LLM‑powered applications and AI‑enabled data platforms.

Employment Terms

• Contractual engagement with a fully remote working arrangement.
• Candidates must be available to work from 6:00 PM to 3:00 AM IST.
• Candidates must be willing to commit to a minimum engagement of one year.
• Immediate joiners or candidates with a short notice period will be preferred.
• Candidates must be comfortable working on a contractual basis and independently owning production data infrastructure.