Purpose

This framework establishes a common language and clear structure for classifying, governing, and consuming data in ARPIA. This terminology is used across the organization for both technical documentation and business communication, ensuring consistency and shared understanding among teams.

---

Data Layers Overview

ARPIA uses a four-layer data architecture to organize data by quality, transformation state, and consumption readiness. Each layer has a defined purpose, color code, and governance rules.

---

Layer Definitions

1. Raw

Definition: Data as it arrives from sources (ERP, POS, IoT, external integrations, files, APIs).

Characteristics:

• No transformation or cleaning applied.
• May contain duplicates, nulls, or format errors.
• Represents the exact "snapshot" of the source.

Examples:

• ventas_raw — sales data with dates as strings and inconsistent fields.
• maestro_sociedades_raw — master data exported directly from SAP.

Color/Tag: Grey ⚪ with tag RAW

---

2. Clean

Definition: Data processed by transformation objects (such as AP Prepared Table or AP SQL Code Execution), validated and standardized.

Note: This layer is equivalent to "Silver" in medallion architecture terminology. You may encounter _silver suffixes in legacy tables — these follow the same standards as _clean tables.

Characteristics:

• Correct data type casting applied.
• Null values handled, duplicates removed.
• Includes clean master data and normalized tables.
• May or may not be included in the Knowledge Grid, but must always be tagged and color-coded.

Examples:

• ventas_clean — sales data with typed dates and normalized costs.
• dim_articulo_clean — article dimension with unified hierarchies.

Color/Tag: Blue 🔵 with tag CLEAN

---

3. Gold

Definition: Data ready for business consumption and inclusion in the Knowledge Grid.

Characteristics:

• Aggregated or transformed into KPIs and business metrics.
• Ambiguity-free: defines the "single source of truth" for sales, budgets, margins, etc.
• Serves as the foundation for both traditional dashboards and AI assistants.

Examples:

• agg_ventas_mes_tienda_gold — monthly sales by store with Net Sales, Budget, Compliance, and Profitability.
• agg_ventas_dia_regional_gold — daily regional sales for operational analysis.

Color/Tag: Gold 🟡 with tag GOLD

---

4. Optimized

Definition: Subset of Clean or Gold data, tailored and optimized for a specific process, application, or consumption pattern.

Characteristics:

• Always documented with its associated Knowledge Node.
• May include indexes, special partitions, or wide structures to accelerate a use case.
• Maintains traceability back to its Clean or Gold origin.

Examples:

• agg_forecast_ventas_optimized — optimized for AI predictive models.
• agg_tienda_dia_dashboard_optimized — optimized for fast executive dashboards.

Color/Tag: Emerald Green 🟢 with tag OPTIMIZED

---

Table Naming Conventions

To ensure tables are easily identifiable by layer, follow these naming patterns:

Prefix Definitions:

• dim_ — Dimension tables (master data, reference data)
• fact_ — Fact tables (transactional data)
• agg_ — Aggregated tables (pre-calculated summaries)

---

How Layers Connect

---

Applying Tags and Colors

Tags

Tags are applied as metadata in the Repository Table settings:

1. Open the table in Repository Tables.
2. Navigate to Settings or Properties.
3. In the Layer Tag field, select the appropriate value: RAW, CLEAN, GOLD, or OPTIMIZED.

Colors

Colors are applied automatically based on the layer tag in:

• Repository table listings
• Data lineage diagrams
• Knowledge Grid visualizations
• Documentation exports

When creating diagrams or documentation manually, use these color codes:

---

Knowledge Grid Integration

The Knowledge Grid is ARPIA's semantic layer that defines how data is exposed to dashboards, reports, and AI assistants. It contains metadata about metrics, hierarchies, joins, and usage policies.

Layer Eligibility:

Knowledge Node: A Knowledge Node is the documentation record for a dataset in the Knowledge Grid. It defines the table's metrics, dimensions, relationships, refresh schedule, and usage policies.

---

Governance Rules

Tagging Requirements

1. All datasets must be tagged with their layer (RAW, CLEAN, GOLD, OPTIMIZED).
2. Official colors must be used in documentation, diagrams, and dashboards to visually reinforce the layer.
3. Table names must include the appropriate suffix (_raw, _clean, _gold, _optimized).

Consumption Rules

Prohibited:

• No dashboard or AI assistant may consume Raw data directly.
• No production reports may reference Clean data without explicit approval.

Responsibilities

---

Visual Flow

Note: Verify this image displays the correct data layer flow diagram.

---

Related Documentation

• Extract & Load
  Learn how data is ingested from sources into the ARPIA platform (Source → Raw).

• Repository Tables
  Explore how loaded data is organized and managed within internal repositories.

• Transform & Prepare
  Continue to the next stage of the ETL lifecycle — cleaning, enriching, and structuring data (Raw → Clean → Gold).

• Knowledge Grid
  Understand how Gold and Optimized data is exposed to dashboards and AI assistants.