P1 - GP - Senior Data Engineer / Data Architect

Atlas Systems

Bengaluru

On-site

INR 4,200,000 - 6,400,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Atlas Systems invites a Senior Data Engineer to build and run end-to-end data pipelines on Databricks, delivering a Gold layer for BI consumption. You will orchestrate Bronze, Silver, and Gold stages, ensuring data quality, governance, and scalable performance across reporting needs.

The role involves collaborating with vendor data teams, reporting and application groups, and finance/portfolio ops to define data models, lineage, and operational runbooks.

Qualifications

  • 5+ years building production data pipelines.
  • 2+ years on Databricks with Delta Lake.
  • Strong SQL and Python/PySpark.
  • Experience with medallion architecture and incremental processing.
  • Familiarity with Unity Catalog and data governance.
  • Able to explain data issues to non-technical users.

Responsibilities

  • Ingestion: build incremental Bronze loads with late-arriving data.
  • Silver: model data with deduplication and type conversions.
  • Gold: implement a business model with dimensions and facts for BI.
  • Migration: support cutover from legacy system and history load.
  • Data modeling: collaborate on item catalog and dimensions.
  • Quality and reconciliation: build checks and regression tests.
  • Serving: publish Gold to downstream consumers and BI.
  • Operations: orchestrate pipelines and manage Unity Catalog access.
  • Collaboration: work with vendor data team and users.

Skills

Databricks Delta Lake
Spark SQL / SQL
Python / PySpark
Data modeling
Data quality
Unity Catalog
Performance tuning
Communication

Tools

Databricks
Unity Catalog
Lakebase
PostgreSQL
Power BI

Job description

About Us:

Atlas Systems Inc. is a software solutions company headquartered in East Brunswick, NJ. Incorporated in 2003, Atlas provides solutions across GRC, Technology, Procurement, Healthcare Provider, and Oracle to customers globally. Atlas helps organizations modernize applications, data platforms, and business operations.

For more information, please visit our website: https://www.atlassystems.com/

Job Description: Senior Data Engineer / Data Architect
About the role

We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.

The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.

What you'll do
  • Ingestion: build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.
  • Silver: model source data close to its native structure, with deterministic deduplication, type‑aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.
  • Gold: implement a source‑agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.
  • Migration: support a single cutover from the legacy system, and load and freeze the history of companies that will not move.
  • Data modeling: work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.
  • Data quality and reconciliation: build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.
  • Serving: publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.
  • Operations: orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.
  • Collaboration: work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.
What you'll bring
  • 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.
  • Strong SQL (Spark SQL and T‑SQL) and Python or PySpark.
  • Hands‑on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.
  • Dimensional modeling: facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.
  • A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.
  • Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.
  • Performance tuning for BI workloads, such as file layout, clustering and query plans.
  • Clear written communication, including explaining data issues and trade‑offs to non‑technical users.
Nice to have
  • Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co‑invest structures.
  • Financial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.
  • Databricks Lakebase or PostgreSQL, including serving data to applications.
  • Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.
  • Azure SQL and migrating SQL Server logic to Databricks.
  • Power BI data modeling, or working closely with BI developers.
  • Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.
Our environment
Area
Tools
Lakehouse

Databricks, Delta Lake, Unity Catalog

Languages

Spark SQL, Python, PySpark, T‑SQL

Application serving

Databricks Lakebase (Postgres) synced tables

Legacy and transition

Azure SQL Database

Reporting

Power BI

Cloud

Microsoft Azure

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Lead Job ID: 411651
Databricks Lead Job ID: 411651

Blumetra Solutions LLC • Hyderabad

Hybrid
INR 1,200,000 - 1,900,000
Lead Data Engineer
Lead Data Engineer

Algoworks • Dadri

On-site
INR 1,200,000 - 2,400,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Bestal • Hyderabad

Hybrid
INR 4,000,000 - 7,000,000
Lead Data Architect (Azure Databricks)
Lead Data Architect (Azure Databricks)

Diacto Technologies • India

Remote
INR 4,000,000 - 7,000,000
Senior Databrics Architect
Senior Databrics Architect

dnA Consulting • Mysuru

On-site
INR 4,000,000 - 7,000,000
Databricks Lead
Databricks Lead

Blumetra Solutions LLC • Hyderabad

Hybrid
INR 4,000,000 - 7,000,000
Senior Databricks Engineer
Senior Databricks Engineer

Value Vision • Bengaluru

On-site
INR 4,200,000 - 6,600,000
Senior Data Engineer
Senior Data Engineer

CoffeeBeans • Bengaluru

On-site
INR 1,800,000 - 2,400,000
IN_Senior Associate_Databricks Data Engineer_GCC_Advisory_Bangalore
IN_Senior Associate_Databricks Data Engineer_GCC_Advisory_Bangalore

Pwc India • Bengaluru

On-site
INR 2,500,000 - 4,000,000
Databricks Developer / Data Architect
Databricks Developer / Data Architect

ExlService Holdings, Inc. • Pune District

Hybrid
INR 1,200,000 - 1,600,000