cnt-collector-node (Cardano Native Tokens Collector Node)

The cnt-collector node is separated into two applications:

• a data collection indexer
• a data submitter that formats price data and submits it to an Orcfax validator node.

The indexer will be run constantly in the background, collecting data and saving it in a local sqlite database, while the submitter node will run from time to time (once every minute for example) and will send the newest collected data to the validator node. In case the indexer application is not running, the submitter will collect the data before submitting it to the validator node, but the process is slow, taking a few minutes, depending on the amount of data that needs to be collected. Submitting correct data to the validator node is more important than submitting quickly data that might be incorrect.

Issues

If you find any iddues with this code, please log them as a new issue under Orcfax Commons.

Quickstart

Most users outside of the Orcfax network will want to run this code and experiment with it without forwaridng data to an Orcfax validator. You can do this as follows.

From pypi

python -m venv venv
source venv/bin/activate
python -m pip install cnt-collector-node

Configure the collector node

An identity file is needed by

The node configuration file (/tmp/.node_identity.json) can be generated by running the script script in node_identity.

python node_identity/create_identity.py

It will generate a json file (/tmp/.node-identity.json) with the following format:

{
  "node_id": "<UUID node id>",
  "location": {
    "ip": "<ip address>",
    "hostname": "<hostname>",
    "city": "<City>",
    "region": "<Region>",
    "country": "<Country code>",
    "loc": "<Latitude and longitude>",
    "org": "<Organisation>",
    "postal": "<Zip code>",
    "timezone": "<Timezone>",
    "readme": "<Link>"
  },
  "initialization": "<Timestamp>",
  "init_version": null,
  "validator_web_socket": "<Websocket address of the validator, like: ws://localhost:8001/ws/node>",
  "validator_certificate": null
}

Environment

Setup the following environment variables (a indexer.env env file is provided to help persist these settings):

export USE_KUPO=true
export OGMIOS_URL=
export KUPO_URL=
export CNT_DB_NAME=

Indexer and submit entry-points

and then run the following for more information:

cnt-indexer --help

or:

cnt-submit --help

Both modes/commands are documented in detail below.

From source

Clone the repository:

git clone git@github.com:orcfax/cnt-collector-node.git
cd cnt-collector-node

Create the python virtual environment and activate it:

python -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt

Configuration and environment

Configure your node and environment as per the pypi instructions.

Run submit

Submit collects live data from the Kupo and Ogmios node and persists price information in the database.

Run submit with:

just submit

The recipe looks as follows;

python -m src.cnt_collector_node.submitter --create-db \
 --identity-file-location /tmp/.node-identity.json \
 --pairs demo_pairs/pairs.py \
 --nopublish

Run index

The indexer indexes CNT data and stores it at CNT_DB_NAME.

Run the indexer:

just index

The recipe looks as follows;

python -m src.cnt_collector_node.indexer --pairs demo_pairs/pairs.py

Inspecting the commands

Both commands can be investigated further by running:

python submitter.py --help

and:

python indexer.py --help

respectively.

Configuring the data sources

The cnt-collector node needs to be configured to know what CNT it needs to collect information about and where to collect the information from.

The configuration lands in the piars.py file in the following format:

{
  "name": "FACT-ADA",
  "token1_policy": "a3931691f5c4e65d01c429e473d0dd24c51afdb6daf88e632a6c1e51",
  "token1_name": "6f7263666178746f6b656e",
  "token1_decimals": 6,
  "token2_policy": "",
  "token2_name": "lovelace",
  "token2_decimals": 6,
  "sources": [
    {
      "source": "MinSwap",
      "address": "addr1z8snz7c4974vzdpxu65ruphl3zjdvtxw8strf2c2tmqnxzf6g882n6sa2gxnk42heavu7uddl5jdl0ektf5f204mmc7s3ykuf9",
      "security_token_policy": "0be55d262b29f564998ff81efe21bdc0022621c12f15af08d0f2ddb1",
      "security_token_name": "b4ba2b47edce71234f328fa20efdb25c3f96e348ca19a683193880489bb368db"
    },
    {
      "source": "WingRiders",
      "address": "addr1z8nvjzjeydcn4atcd93aac8allvrpjn7pjr2qsweukpnayg6pp9snyy9v7uwarxd7dqc5k52egtc49y5w5h3nqqdy6qs2nzs8y",
      "security_token_policy": "026a18d04a0c642759bb3d83b12e3344894e5c1c7b2aeb1a2113a570",
      "security_token_name": "4c"
    },
    {
      "source": "SundaeSwap",
      "address": "addr1w9qzpelu9hn45pefc0xr4ac4kdxeswq7pndul2vuj59u8tqaxdznu",
      "security_token_policy": "0029cb7c88c7567b63d1a512c0ed626aa169688ec980730c0473b913",
      "security_token_name": "7020fb04"
    },
    {
      "source": "Spectrum",
      "address": "addr1x94ec3t25egvhqy2n265xfhq882jxhkknurfe9ny4rl9k6dj764lvrxdayh2ux30fl0ktuh27csgmpevdu89jlxppvrst84slu",
      "security_token_policy": "d740d4088886a6b7e4ab8293424308515590e93e826cb874f8c92aff",
      "security_token_name": "6f7263666178746f6b656e5f4144415f4e4654"
    }
  ]
}

A full demo example is in the demo_pairs directory.

This config file is generated automatically by running the code in cnt-collector-config.

Config details

Each UTxO with a liquidity pool (a token pair in a DEX) contains a unique token (NFT) which authenticates the liquidity pool. This token is used to identify each liquidity pool. However, it's important to note that the smart contract address for a liquidity pool from a DEX can change from time to time (eg if the DEX wants to change it in order to delegate it to a stake pool). This means that the CNT configured in pairs.py may require occasional updating.

In a liquidity pool, each token of the pair has 50% of the pool's total value-- making each equivalent in value. This means that the price of the target token can be calculated by dividing the numbers of tokens in the pair.

Example: if the liquidity pool ADA/TokenX has 50k ADA and 350k TokenX, then the price of TokenX in ADA is:

price = 50000 / 350000 = 0.142857143

When calculating the average price on all DEXes, the total amount of available tokens in the CNT on all DEXes is taken into account. From each DEX, the largest liquidity pool is taken into account, because the others are much smaller and they are irrelevant (this needs to be checked from time to time, to make sure it doesn't change).

The pairs.py file included in this repository is a correct configuration file currently, it can be used for testing purposes. For production, it's a good idea to generate it again.

Indexer

The indexer script (indexer.py) connects to Kupo and Ogmios to read the configured DEX liquidity pools data and save it into a sqlite database table for the collector script.

The table is called "utxos" and has the following structure:

CREATE TABLE utxos (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    pair TEXT NOT NULL,
    source TEXT NOT NULL,
    price FLOAT NOT NULL,
    block_height INTEGER NOT NULL,
    address TEXT NOT NULL,
    token1_policy TEXT NOT NULL,
    token1_name TEXT NOT NULL,
    token1_decimals INTEGER NOT NULL,
    token2_policy TEXT NOT NULL,
    token2_name TEXT NOT NULL,
    token2_decimals INTEGER NOT NULL,
    security_token_policy TEXT NOT NULL,
    security_token_name TEXT NOT NULL,
    token1_amount INTEGER NOT NULL,
    token2_amount INTEGER NOT NULL,
    tx_hash TEXT NOT NULL,
    output_index INTEGER NOT NULL,
    date_time timestamp
);

The indexer should run continuously. There are 2 threads:

1. the populate_utxos threads, which inserts or updates the data in the utxos table. it runs when the script starts and after that every UTXOS_THREAD_TIMEOUT seconds (configurable in config.py)
2. the main execution thread, which connects to Ogmios and requests all the new blocks created in real time. it parses each transaction from each block, and if a transaction is updating the UTxO of a liquidity pair configured in pairs.py, it updates the utxo record for that pair on that DEX in the utxos table and inserts a new data point into the price table.

The data points saved in the price table is not used when submitting the data to the validator node. It is saved for archiving and troubleshooting purposes.

The table where the datapoints are saved has the following format:

CREATE TABLE IF NOT EXISTS price (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    pair TEXT NOT NULL,
    source TEXT NOT NULL,
    price FLOAT NOT NULL,
    token1_amount INTEGER NOT NULL,
    token2_amount INTEGER NOT NULL,
    epoch INTEGER NOT NULL,
    block_height INTEGER NOT NULL,
    date_time timestamp
);

Each update of an UTxO in the database table and each new block received from Ogmios also triggers an update of the status table, which keeps track of the latest block slot in the blockchain (and the timestamp when the record was updated).

CREATE TABLE status (
    id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    current_block_slot INTEGER NOT NULL,
    date_time timestamp
    );

Submit

The script (submitter.py) calculates the prices of the configured CNT pairs from Cardano DEX liquidity pools and sends the prices to the configured validator node.

The script should be run at regular intervals, and it will save a datapoint for each pair of tokens (in each DEX) in a database.

The script needs to connect to an Ogmios server to read the blockchain data.

As a significant improvement, the collector script does not read the liquidity pools data directly from the Ogmios, but from the sqlite database created by the indexer script. Only when the data in the indexer script is outdated (which is detected by reading the data saved in the status table and comparing it with the data read from Ogmios), the script will read the data directly from Ogmios.

Justfile

A justfile is included in the rerpo for convenience functions. See just for more information on how to install this.