Carol

Integrate your App

This chapter describes the steps to develop a native integration into Carol.

You can download the Carol Cloud (2C) application (github) to take a look at how these services work.

1. Swagger

Carol has Swagger embedded, meaning that you can call the REST services using Swagger to quickly test the REST APIs. The URL to get access to Swagger is: https://<tenant>.carol.ai/swagger-ui/.

All services in Swagger have basic documentation allowing the user to understand each service (when applicable, including a Json sample to call the service).

If you feel that the documentation could be improved, please contact us: [email protected]

By default, swagger will show a very minimized list of services, available for users not authenticated.

To open the full list of services available, you should do the login process.

Carol: Services before authentication

Carol: Services before authentication

2. Authentication

Carol has authentication following the specification OAuth 2.0, the grant_type allowed today is password and thus username and password should be provided for the authentication process. Soon others grant_type will be allowed.

The service v2 OAuth allows the user to complete authentication and refresh the access_token to renew the actual session.

These are the most important services available on v2 OAuth services:

Carol: Authentication services

Carol: Authentication services

A brief description of each service is as follows:

Service
Description

GET /api/v2/oauth2/token/{access_token}

Service to get the access_token details (expiration)

POST /api/v2/oauth2/token

Service to authenticate the user, more details below

POST /api/v2/oauth2/token

Service to authenticate the user, the following parameters are mandatory:

Parameter
Value

grant_type

password or refresh_token.

username

as defined when the user was created.

password

as defined when created the user.

refresh_token

Token to refresh the session, required when grant type is refresh_token.

subdomain

as defined when the tenant was created

connectorId

as defined in Carol for this connector

This is a sample to demonstrate the authentication process:

Carol: Authentication response

Carol: Authentication response

Considering the result below, this is the most important information to be considered:

Variable
Description

refresh_token

Used when refreshing the token (when the session is about to expire)

access_token

Used on the Authorization request header

expires_in

Time left for the current session to expire, should be controlled to refresh this session

To refresh the session, you should call the service token again with the parameters below:

Parameter
Value

grant_type

refresh_token

refresh_token

the value for refresh_token received when the user was authenticated

When calling the token service to refresh the session, the result will be the same as when authenticating the user. At this time a new access_token will be provided.

Since we have the access_token, we are able to start calling other services. Swagger has a field API Key that accepts the access_token to authenticate the next calls:

Carol: Authentication in Swagger

Carol: Authentication in Swagger

3. Sending Data

Before start sending the data, I strongly recommend you to read the concepts in Carol: Getting Started with Carol.

The service to send the schema and the service to send data are the most important services in v2 Staging services. These services allow you to send any kind of data to Carol.

These are the services available on v2 Staging:

Carol: Staging services

Carol: Staging services

DELETE /api/v2/staging/tables/{table}/documents

Service to remove all data for a specific staging table in the Staging Area.

GET /api/v2/staging/tables/{table}/schema

Service to get the schema sent to a specific staging table.

GET /api/v2/staging/tables/{table}/{id}

Service to get a specific record from the staging area by ID. An important point is that the staging area is a temporary location before generating the golden records. While generating the golden records, the staging area will be removed.

POST /api/v2/staging/tables/{table}

Service to send data to a specific staging table. It is mandatory define the schema before sending the data.

This service allows sending the data in a gzip format, saving a huge quantity of time during the sending process.

Swagger has a complete sample on how to call this service.

Ps.: Each request is limited a 5 MB of data. It means that you should split the request in a partial quantity of data to fit in 5 MB.

POST /api/v2/staging/tables/{table}/restore

Service to restore staging area backup to reprocess the data for a specific staging table.

POST /api/v2/staging/tables/{table}/schema

This service is to create the schema for a specific staging table. This is a requirement to start sending data to Carol.

The schema can be defined as flexible, it means that the schema will change over time and Carol will automatically detect these changes. New fields can be added, or the field type can be changed (from Integer to String, for example). These changes are very common in the noSQL environment (like MongoDB).

These are the types currently supported in the Staging Area: STRING, BOOLEAN, LONG, DOUBLE, DATE, NESTED, OBJECT and GEOPOINT.

The attribute "mdmCrossreference" define the unique keys for this staging table.

Json Schema:

{
    "mdmFlexible": true,
    "mdmCrosswalkTemplate": {
        "mdmCrossreference": {
            "emitente": [
                "cod-emitente"
            ]
        }
    },
    "mdmStagingMapping": {
        "properties": {
            "cgc": {
                "type": "string"
            },
            "nome-emit": {
                "type": "string"
            },
            "cod-emitente": {
                "type": "integer"
            },
            "telefones": {
                "type": "object",
                "properties": {
                    "tipo": {
                      "type": "string"
                    },
                    "telefone": {
                      "type": "string"
                    }
                 }
             }
        }
    }
}

POST /api/v2/staging/restore

Service to restore staging area backup to reprocess the entire staging area (all staging tables for this connector).

POST /api/v2/staging/search

This service is used to searches in Carol using SCIM queries, you can find more details about the ways to consume data (here).

This is an sample of how to query data:

The SCIM query below search all documents that have data to search in mdmhealthcareprovidertaxonomycode field or mdmhealthcareprovidertaxonomydescription field.

nested mdmGoldenFieldAndValues (mdmGoldenFieldAndValues.mdmhealthcareprovidertaxonomycode co "data to search" OR mdmGoldenFieldAndValues.mdmhealthcareprovidertaxonomydescription co "data to search"

This is one way to consume data and normally will be used for applications that do not have the database, like mobile applications. This kind of application will use Carol as a database.

Ps.: This service is deprecated. We strongly recommend using the service "Filter" or "namedQuery"

PUT /api/v2/staging/tables/{table}/schema

This service is used to update the schema for a staging table. The json schema to be sent should be complete, following the same structure when called the service to define the schema.

4. Consuming Data

To start consuming data, first, you have to allow the application to consume a specific data model.

The process to consume golden records allows the connector to consume in two different formats:

  • Golden Record schema: in this way, Carol will send the records following the schema defined for this specific data model.
  • Connector Schema: in this way, Carol will send the records following the schema used to send data to Carol.

Both ways to consume data from Carol allow us to add transformation rules to transform the data when consuming records.

The next image shows all data consumed by a specific connector (Bemacash) indicated by the number "1" and it allows the user to add a new "Consumer" (data to be consumed) to allow this connector to consume other golden records.

Carol: Consumption process

Carol: Consumption process

Clicking to add a new consumer, you should specify the format of the Golden Record you want to consume (remembering, it can be following the data model format or the format that this connector sent the records).

Carol: Consumption format

Carol: Consumption format

After completing these steps, the application is ready to start consuming golden records from Carol. The next step will be to call the REST services to consume data.

Regardless of the consumption process choice (Data Model schema or Connector schema), Carol allows the reverse transformation to be applied, meaning that Carol allows rules to be applied to change the value to fit the needs of the consumer application.

Services to Consume Data

There are 4 services that allow us to consume data from Carol, the below figure shows all services available.

Carol: services for consumption

Carol: services for consumption

GET /api/v2/data_consumption/records

This service returns golden records to be consumed. This service will consume golden records in the queue for this specific application (connectorId sent on login process). This service requires these parameters:

Parameter
Description

entityList

Array with all dataModelIds and the start counter to indicate which is the first record.

pageSize

Indicates how many records will be consumed, the limit is 1000 records per request

A sample for entityList:

[{"entityId":"5f8d18c0c92a11e580ae7608311998e9", "startCounter": -1}, {"entityId":"24237b70c92b11e580ae7608311998e9", "startCounter": 9}]

This sample is requesting to consume golden records from 2 different data models. The first entity starts the counter from -1 and the second starts the counter from 9.

The counter indicates the sequence to be consumed in the queue for this application. The counter -1 means to send the next records (not confirmed) in the queue.

After consuming golden records, the application should confirm that the golden record was consumed, with this Carol will remove this golden record from the queue for this connector to avoid consuming the record twice.

The image below shows the process to start mapping the consumption process. If this connector is contributing (sending data) Carol will automatically map the consumption model and the data model considering the initial mapping (staging area to data model).

Connector: Number of records pending for consumption

Connector: Number of records pending for consumption

Carol allows the mapping and add transformation rules to be changed (normally used to convert data, apply substring operation, apply rules to convert the data, etc) everything to make the consumption process on the connector side easier.

Carol: transformation rules for consumption process

Carol: transformation rules for consumption process

The next topics will describe the services available to start consuming golden records.

GET /api/v2/data_consumption/{dataModelId}/using_ids

This service is responsible for returning golden records with a specific Golden Record IDs. The ID is an internal identification for all records in Carol.

The parameters to call this service:

Parameter
Description

dataModelId

Entity to be consumed

mdmIds

Array with all Golden Record IDs to be consumed

This service will not remove the records from the application queue.

GET /api/v2/data_consumption/{dataModelId}/using_identifier

This service is responsible for returning golden records with a specific identifier/crosswalk (primary key defined in the staging table). The identifier/crosswalk is the identifier, from the connector side, for a specific record in a specific staging table.

The parameters to call this service:

Parameter
Description

dataModelId

Entity to be consumed

connectorId

Application that sent the record

stagingTable

The staging table that will have the crosswalk

crosswalks

crosswalk/identifier (or primary key) to search

This is a sample of a crosswalk/identifier:

[{"emitente":{"cod-emitente":"5301"}}, {"emitente":{"cod-emitente":"4562"}}]

The crosswalk above searches for two different records in emitente staging table by the identifier cod-emitente.

This service will not remove the records from the application queue.

POST /api/v2/data_consumption/confirm_ids

This service is responsible for confirming a consumption process. After calling the service to consume the records and update correctly the record in the local environment (customer application side), the connector should confirm the consumption to remove this record from the next consumption process.

The parameters to call this service:

Parameter
Description

body

Json to inform the DataModelId and Golden Record IDs consumed.

This is a sample of confirming records consumed:

{"5aedr24456542d5f32e":["6ecf2ed0a32411e5b2b7da33014e29de", "4ecf5ed453241133b2b7da33014e29de"]}

This service allows one to confirm more than one record per request, making the confirmation process faster.

5. Golden Record Details

After consuming a record, the connector should get the record and update locally. This section will describe details about the golden record structure.

The golden record has this structure:

{
  "9a22a090ff0811e59f170401c8804501": {
    "goldenRecords": [   
      {
        "mdmEntityTemplateId": "9a22a090ff0811e59f170401c8804501",
        "mdmGoldenFieldAndValues": { 
          "mdmprovidertypecode": "1",
          "mdmproviderotherlegalbusinesstypecode": "",
          "mdmauthorizedofficialtitleposition": "",
          "mdmproviderothercredential": "",
          "mdmhealthcareprovidertaxonomycode": "2080P0207X",
          "mdmproviderothernameprefix": "",
          "mdmhealthcareproviderprimarytaxonomy": "Y",
          "mdmnpi": "1831192947",
          "mdmproviderotherlegalbusinessname": "",
          "mdmaddressid": [
            "1275b5b0ff0d11e592210401c8804501",
            "1279fb70ff0d11e592210401c8804501"
          ],
          "mdmauthorizedofficialfirstname": "",
          "mdmproviderlicensenumberstate": "OH",
          "mdmauthorizedofficiallastname": "",
          "mdmproviderothernamesuffix": "",
          "mdmproviderotherlastnametypecode": "",
          "mdmproviderothermiddlename": "",
          "mdmphone": [
            {
              "mdmphonenumber": "6147223552"
            },
            {
              "mdmphonenumber": "6147223552"
            }
          ],
          "mdmprovidernameprefix": "DR.",
          "mdmprovidermiddlename": "M",
          "mdmproviderlicensenumber": "35123299",
          "mdmproviderotherlastname": "",
          "mdmein": "",
          "mdmpartnerdate": "",
          "mdmprovidernamesuffix": "",
          "mdmreplacementnpi": "",
          "mdmprovidergender": "M",
          "mdmhealthcareprovidertaxonomygroup": "",
          "mdmauthorizedofficialmiddlename": "",
          "mdmprovidercredential": "MD, PHD",
          "mdmproviderlastname": "HORWITZ",
          "mdmproviderlegalbusinessname": "",
          "mdmproviderfirstname": "EDWIN",
          "mdmhealthcareproviderotherinfo": [
            {
              "mdmotherprovideridentifiertypecode": "02",
              "mdmotherprovideridentifier": "G51401",
              "otherprovideridentifierissuer": "",
              "otherprovideridentifierstate": "TN"
            },
            {
              "mdmotherprovideridentifiertypecode": "05",
              "mdmotherprovideridentifier": "422400000",
              "otherprovideridentifierissuer": "",
              "otherprovideridentifierstate": "ME"
            },
            {
              "mdmotherprovideridentifiertypecode": "",
              "mdmotherprovideridentifier": "",
              "otherprovideridentifierissuer": "",
              "otherprovideridentifierstate": ""
            }
          ],
          "mdmproviderfullname": "EDWIN M HORWITZ",
          "mdmproviderotherfirstname": "",
          "mdmauthorizedofficialphone": ""
        },
        "mdmCrosswalk": [ 
          {
            "mdmDataSourceId": "d5b15a958628c61087fbce66d329f01d",
            "mdmCrossreference": {
              "nppes": {
                "npi": "1831192947"
              }
            }
          }
        ],
        "mdmCounterForEntity": 1645,
        "mdmProfileTitle": "DR. EDWIN M HORWITZ  MD, PHD",
        "mdmId": "13009090ff0d11e592210401c8804501",
        "mdmElasticsearchMappingType": "mdmhcpGolden",
        "mdmCreated": "2016-04-10T11:12:12.083Z",
        "mdmLastUpdated": "2016-04-10T11:12:12.315Z",
        "mdmTenantId": "f1dd24df6e0a837871e49083d1a5fc66"
      },
      {
        "mdmEntityTemplateId": "9a22a090ff0811e59f170401c8804501",
        "mdmGoldenFieldAndValues": {
          "mdmprovidertypecode": "1",
          "mdmproviderotherlegalbusinesstypecode": "",
          "mdmauthorizedofficialtitleposition": "",
          "mdmproviderothercredential": "",
          "mdmhealthcareprovidertaxonomycode": "207X00000X",
          "mdmproviderothernameprefix": "",
         }
        ],
        "mdmCounterForEntity": 1656,
        "mdmProfileTitle": "DR. ANIL S.A. DANIVAS  ",
        "mdmId": "13cc6c10ff0d11e592210401c8804501",
        "mdmElasticsearchMappingType": "mdmhcpGolden",
        "mdmCreated": "2016-04-10T11:12:13.477Z",
        "mdmLastUpdated": "2016-04-10T11:12:13.651Z",
        "mdmTenantId": "f1dd24df6e0a837871e49083d1a5fc66"
      }
    ],
    "numOfPendingRecords": 3602968, 
    "lastCounterForEntity": 3615810,
    "lastCounterConsumed": 1656,
    "message": ""
  }
}

Golden record details:

Field
Description

9a22a090ff0811e59f170401c8804501

Data Model ID (mdmEntityTemplateId) consumed

goldenRecords

Array of golden records consumed

mdmGoldenFieldAndValues

Attributes related to one golden record. This fields will change considering the schema that you are consuming (Data Model schema or Connector schema)

mdmCrosswalk

Connector identifier. With this information, the consumer will identify if this data was sent by this connector and if it is just an update process or if it is a new record and should insert this record.

numOfPendingRecords

Total number of records pending.