Links

For in depth discussion of this feature please see:

https://confluence.ccs.ornl.gov/spaces/DATAFED/pages/442081697/GENERAL+-+Metadata+only+Repository

Description

There is a need to allow creation of metadata only allocations, meaning storage for raw data files would only be supported externally. There is also interest in supporting other types of allocations on different storage mediums other than POSIX filesystems. Any rework of the repository and allocation types would need to be flexible enough to accomodate future changes along the same lines.

Opinionated Design Approach

If our goal is to make as few changes as possible as first pass for an MVP, then we will resort to setting reasonable defaults instead of adding and removing items from the documents.

Repository Document

Below is an example of a repo as stored in the database.

{
    "_key" : "empty",
    "_id" : "repo/empty",
    "capacity" : 80000000000000,
    "pub_key" : "dummy_repo_key",
    "address" : "tcp://somedomain:9000",
    "endpoint" : "<endpoint UUID>",
    "path" : "/path/to/content/empty/",
    "title" : "My dummy repo",
    "desc" : "",
    "domain" : "localhost"
}

Notice, for a metatadata only repository the following fields are not really needed:

• capacity - there is no data capacity specified because there are no raw data resources allocated, record limits are handled on the allocation level.
• pub_key - this is only needed for encrypted communication between the repository which there isn't one.
• address - address to the repository, this is not used because all of the metadata is in the metadata database
• endpoint - globus endpoint uuid, not used
• path - posix file path on the repo where the data is located again not used
• title - this is still valid
• desc - this is still valid
• domain - I'm actually not sure what this is used for

The only items that are really useful are:

• _key
• _id
• title
• desc

The other items could simply be given reasonable defaults

• capacity: 0
• pub_key: ""
• address: ""
• endpoint: ""
• path: ""
• domain: ""

Allocation Document

Below is an example of the data stored in an allocation collection in the database.

{
    "_key" : "6956825",
    "_id" : "alloc/6956825",
    "_from" : "u/<someone>_username",
    "_to" : "repo/cades-test",
    "_rev" : "_iAbn6Eq--t",
    "path" : "/data/cades-test/user/<someone>_username/",
    "rec_count" : 0,
    "rec_limit" : 1000,
    "data_size" : 0,
    "data_limit" : 104857600
}

For metadata only repositories, the following fields are not needed.

• data_size - there is no raw data attached to the repository
• data_limit - there is no raw data attached to the repository so there is no limit

Both of these values could reasonably be set to having a value of 0.

In terms of implementation, the repo_router.js is not making use of the repo.js object, it would make sense as part of this feature to change the implementation of the repo_router.js to use the repo.js class. Reusable logic could be made part of that class. The repo.js class also likely needs to be moved into the models folder in core/database/foxx/api/models

Files impacted

Foxx

• core/database/foxx/api/repo_router.js
• core/database/foxx/api/repo.js

Core Service

• core/server/ClientWorker.cpp

Common library

• common/proto/common/SDMS_Auth.proto