Off-chain Indexing

There are times when on-chain extrinsics need to pass data to the off-chain worker context with predictable write behavior. We can surely pass this piece of data via on-chain storage, but this is costly and it will make the data propagate among the blockchain network. If this is not a piece of information that need to be saved on-chain, another way is to save this data in off-chain local storage via off-chain indexing.

As off-chain indexing is called in on-chain context, if it is agreed upon by the blockchain consensus mechanism, then it is expected to run predictably by all nodes in the network. One use case is to store only the hash of certain information in on-chain storage for verification purpose but keeping the full data set off-chain for lookup later. In this case the original data can be saved via off-chain indexing.

Notice as off-chain indexing is called and data is saved on every block import (this also includes forks), the consequence is that in case non-unique keys are used the data might be overwritten by different forked blocks and the content of off-chain database will be different between nodes. Care should be taken in choosing the right indexing key to prevent potential overwrites if not desired.

We will demonstrate this in ocw-demo pallet. Knowledge discussed in this chapter built upon using local storage in off-chain worker context.

Notes

In order to see the off-chain indexing feature in effect, please run the kitchen node with off-chain indexing flag on, as ./target/release/kitchen-node --dev --tmp --enable-offchain-indexing true

Writing to Off-chain Storage From On-chain Context

src: pallets/ocw-demo/src/lib.rs


#![allow(unused)]
fn main() {
#[derive(Debug, Deserialize, Encode, Decode, Default)]
struct IndexingData(Vec<u8>, u64);

const ONCHAIN_TX_KEY: &[u8] = b"ocw-demo::storage::tx";

// -- snip --

pub fn submit_number_signed(origin, number: u64) -> DispatchResult {
  // -- snip --
  let key = Self::derived_key(frame_system::Module::<T>::block_number());
  let data = IndexingData(b"submit_number_unsigned".to_vec(), number);
  offchain_index::set(&key, &data.encode());
}

impl<T: Config> Module<T> {
  fn derived_key(block_number: T::BlockNumber) -> Vec<u8> {
    block_number.using_encoded(|encoded_bn| {
      ONCHAIN_TX_KEY.clone().into_iter()
        .chain(b"/".into_iter())
        .chain(encoded_bn)
        .copied()
        .collect::<Vec<u8>>()
    })
  }
}
}

We first define a key used in the local off-chain storage. It is formed in the derive_key function that append an encoded block number to a pre-defined prefix. Then we write to the storage with offchain_index::set(key, value) function. Here offchain_index::set() accepts values in byte format (&[u8]) so we encode the data structure IndexingData first. If you refer back to offchain_index API rustdoc, you will see there are only set() and clear() functions. This means from the on-chain context, we only expect to write to this local off-chain storage location but not reading from it, and we cannot pass data within on-chain context using this method.

Reading the Data in Off-chain Context

src: pallets/ocw-demo/src/lib.rs


#![allow(unused)]
fn main() {
fn offchain_worker(block_number: T::BlockNumber) {
  // -- snip --

  // Reading back the off-chain indexing value. It is exactly the same as reading from
  // ocw local storage.
  let key = Self::derived_key(block_number);
  let oci_mem = StorageValueRef::persistent(&key);

  if let Some(Some(data)) = oci_mem.get::<IndexingData>() {
    debug::info!("off-chain indexing data: {:?}, {:?}",
      str::from_utf8(&data.0).unwrap_or("error"), data.1);
  } else {
    debug::info!("no off-chain indexing data retrieved.");
  }

  // -- snip --
}
}

We read the data back in the offchain_worker() function as we would normally read from the local off-chain storage. We first specify the memory space with StorageValueRef::persistent() with its key, and then read back the data with get and decode it to IndexingData.

Reference