ADR 062: Collections, a simplified storage layer for cosmos-sdk modules.
- 30/11/2022: PROPOSED
PROPOSED - Implemented
We propose a simplified module storage layer which leverages golang generics to allow module developers to handle module storage in a simple and straightforward manner, whilst offering safety, extensibility and standardisation.
Module developers are forced into manually implementing storage functionalities in their modules, those functionalities include but are not limited to:
- Defining key to bytes formats.
- Defining value to bytes formats.
- Defining secondary indexes.
- Defining query methods to expose outside to deal with storage.
- Defining local methods to deal with storage writing.
- Dealing with genesis imports and exports.
- Writing tests for all the above.
This brings in a lot of problems:
- It blocks developers from focusing on the most important part: writing business logic.
- Key to bytes formats are complex and their definition is error-prone, for example:
- how do I format time to bytes in such a way that bytes are sorted?
- how do I ensure when I don't have namespace collisions when dealing with secondary indexes?
- The lack of standardisation makes life hard for clients, and the problem is exacerbated when it comes to providing proofs for objects present in state. Clients are forced to maintain a list of object paths to gather proofs.
Current Solution: ORM
The current SDK proposed solution to this problem is ORM. Whilst ORM offers a lot of good functionality aimed at solving these specific problems, it has some downsides:
- It requires migrations.
- It uses the newest protobuf golang API, whilst the SDK still mainly uses gogoproto.
- Integrating ORM into a module would require the developer to deal with two different golang frameworks (golang protobuf + gogoproto) representing the same API objects.
- It has a high learning curve, even for simple storage layers as it requires developers to have knowledge around protobuf options, custom cosmos-sdk storage extensions, and tooling download. Then after this they still need to learn the code-generated API.
CosmWasm Solution: cw-storage-plus
The collections API takes inspiration from cw-storage-plus, which has demonstrated to be a powerful tool for dealing with storage in CosmWasm contracts. It's simple, does not require extra tooling, it makes it easy to deal with complex storage structures (indexes, snapshot, etc). The API is straightforward and explicit.
We propose to port the
collections API, whose implementation lives in NibiruChain/collections to cosmos-sdk.
Collections implements four different storage handlers types:
Map: which deals with simple
KeySet: which acts as a
Setand only retains keys and no object (usecase: allow-lists).
Item: which always contains only one object (usecase: Params)
Sequence: which implements a simple always increasing number (usecase: Nonces)
IndexedMap: builds on top of
KeySetand allows to create relationships with
All the collection APIs build on top of the simple
Collections is fully generic, meaning that anything can be used as
Value. It can be a protobuf object or not.
Collections types, in fact, delegate the duty of serialisation of keys and values to a secondary collections API component called
ValueEncoders take care of converting a value to bytes (relevant only for
Map). And offers a plug and play layer which allows us to change how we encode objects,
which is relevant for swapping serialisation frameworks and enhancing performance.
Collections already comes in with default
ValueEncoders, specifically for: protobuf objects, special SDK types (sdk.Int, sdk.Dec).
KeyEncoders take care of converting keys to bytes,
collections already comes in with some default
KeyEncoders for some privimite golang types
(uint64, string, time.Time, ...) and some widely used sdk types (sdk.Acc/Val/ConsAddress, sdk.Int/Dec, ...).
These default implementations also offer safety around proper lexicographic ordering and namespace-collision.
Examples of the collections API can be found here:
- introduction: https://github.com/NibiruChain/collections/tree/main/examples
- usage in nibiru: x/oracle, x/perp
- cosmos-sdk's x/staking migrated: https://github.com/testinginprod/cosmos-sdk/pull/22
The design of
KeyEncoders allows modules to retain the same
byte(key)=>byte(value) mappings, making
the upgrade to the new storage layer non-state breaking.
- ADR aimed at removing code from the SDK rather than adding it. Migrating just
x/stakingto collections would yield to a net decrease in LOC (even considering the addition of collections itself).
- Simplifies and standardises storage layers across modules in the SDK.
- Does not require to have to deal with protobuf.
- It's pure golang code.
- Generalisation over
ValueEncodersallows us to not tie ourself to the data serialisation framework.
ValueEncoderscan be extended to provide schema reflection.
- Golang generics are not as battle-tested as other Golang features, despite being used in production right now.
- Collection types instantiation needs to be improved.
- Automatic genesis import/export (not implemented because of API breakage)
- Schema reflection