Golden Record View

This chapter describes how Golden Record View works in Carol.

Objective

The main objective of Golden Record View is to combine data from two Data Model in a unified record.

Prerequisites

In order to create a Golden Record View you will need the following configuration:

  • Two Data Models
  • Relationship between Data Models
  • Relationship View between Data Models

Relationship View

Basically, a Relationship View is the resource that defines how the Golden Record View is going to be configured.

Example of Usage

For this example we will use the Data Models Product and Purchase Orders.

Create Data Models Relationship

In our example we will create a relationship between Product and Purchase Orders

👍

Relationship Fact Table

Be aware that when you create the relationship the Data Model that you are using to create the relationship is always consider to be the Relationship Fact Table and the Golden Record View is create using LEFT OUTER JOIN.

In our example the Purchase Orders will be our Fact Table and the Golden Record View will be created performing LEFT OUTER JOIN on Product.

14031403

Relationship between Purchase Orders and Product

Create RelationshipView

In order to create a Golden Record View we need to create a RelationshipView that defines how the Golden Record View will be configured. In our example we want to join data from Purchase Orders and Product.

📘

VIEW Index

Once the Relationship View is created, a new index called VIEW is created on Elasticsearch. This new index is where all the Golden Records View are going to be created with the EntityType equals to the name of the Relation Ship View.

To create a RelationshipView you need to call the following endpoint:
POST /api/v1/relationshipView

14281428

POST /api/v1/relationshipView

{
  "mdmActions":[],
  "mdmName":"has_product_view",
  "mdmRelationshipMappingId":"80dd8770613a11e9ba31acde48001122",
  "mdmRelationshipViewSourceEntityInputField":[
     "mdmorderid",
     "mdmcustomername",
     "mdmcustomerid",
     "mdmtotalvalue"
  ],
  "mdmRelationshipViewTargetEntityInputField":[
     "mdmname",
     "mdmdescription",
     "mdmproducttype"
  ]
}
AttributeDescription
mdmActionsAction that will be performed during the creation of the GoldenRecordView
mdmNameThis is the name of the Relationship View which also will be the EntityType for the Golden Record View for query purpose
mdmRelationshipMappingIdThis is the Id of the Relationship that was defined on previous step.
mdmRelationshipViewSourceEntityInputFieldHere you need to define the fields that you want to bring to your Golden Record View. The fields that you will define here needs to be part of your Fact Table Data Model defined on the Relationship.
mdmRelationshipViewTargetEntityInputFieldHere you need to define the fields that you want to bring to your Golden Record View. The fields that you will define here needs to be part of your other Data Model defined on the Relationship.

mdmActions

📘

Types of Operations with mdmActions

At this moment the supported operations on mdmActions are: SUM and AVERAGE . Also the actions are performed on only one field.

{
      "mdmActions": [
        {
          "inputFieldname": "mdmunitvalue",
          "operation": "SUM",
          "origin": "TARGET",
          "outputFieldname": "purchase_totalvalue"
        },
        {
          "inputFieldname": "mdmunitvalue",
          "operation": "AVERAGE",
          "origin": "TARGET",
          "outputFieldname": "purchase_average"
        }
      ],
      "mdmCreated": "2019-07-23T20:00:41.660Z",
      "mdmCreatedUser": "[email protected]",
      "mdmEntityType": "mdmRelationshipView",
      "mdmId": "8c7d9700ad8411e9a9e5acde48001122",
      "mdmLastUpdated": "2019-07-23T20:00:41.660Z",
      "mdmName": "has_product_view",
      "mdmRelationshipMappingId": "6f282120ad8411e9a9e5acde48001122",
      "mdmRelationshipViewSourceEntityInputField": [
        "mdmcustomername",
        "mdmorderid"
      ],
      "mdmRelationshipViewTargetEntityInputField": [
        "mdmunitvalue",
        "mdmname"
      ],
      "mdmRunningState": "RUNNING",
      "mdmTenantId": "e4010da4110ba377d100f050cb4440db",
      "mdmUpdatedUser": "[email protected]"
    }
AttributeDescription
inputFieldnameName of the field that will be the input value
operationPossible values: SUM OR AVERAGE
originIndicate where the input field is located in reference to the relationship. Possible values: SOURCE OR TARGET. Default TARGET
outputFieldnameName of the attribute that will be appended to the GoldenRecordView with the result of the operation

Manage Golden Record View

📘

Automatically Process

When the Relationship View is created ANY interaction with a Golden Record that belongs to the Data Models Purchase Orders and Product will interact with the Golden Record View.

You can also reprocess all your Golden Record View without touching the Golden Records using the
POST /api/v1/goldenRecordView/reprocess endpoint.

There are three parameters on this API, and they are independent as described below:

ParameterDescription
entityTemplateIdWhen defined, the application will search for this entity template id and find all the relationship mapping that this entity template may have. Once the application knows every relationship mapping for this entity template them it will search all the relationship view for every relationship mapping. Which means, that if you define the entityTemplateId, all the Golden Record View that has direct or indirect reference with this entity template will be reprocessed.
relationshipMappingIdWhen defined, the application will search for all the relationship view that belongs to this relationship mapping. Which means, that if you define the relationshipMappingId, all the Golden Record View that has direct or indirect reference with this relationshipMappingId will be reprocessed.
relationshipViewIdWhen defined, the application will reprocess all the Golden Record View that belongs to this relationshipViewId.
14291429

Once this endpoint to reprocess the Golden Record View is executed a new task is create and you can see the progress on the UI.

14071407

Query Golden Record View

You can also search and query the Golden Record View using the POST /api/v2/queries/filter endpoint

12841284
{
  "mustList": [
    {
      "mdmFilterType": "TYPE_FILTER",
      "mdmValue": "has_product_view"
    }
  ],
  "shouldList": [
    {
      "mdmFilterType": "TERM_FILTER",
      "mdmKey": "mdmGoldenFieldAndValues.mdmorderid.raw",
      "mdmValue": "81744547"
    }
  ]
}