Encoding
While encoding in the Cosmos SDK used to be mainly handled by go-amino
codec, the Cosmos SDK is moving towards using gogoprotobuf
for both state and client-side encoding.
Encoding
The Cosmos SDK utilizes two binary wire encoding protocols, Amino which is an object encoding specification and Protocol Buffers, a subset of Proto3 with an extension for interface support. See the Proto3 spec for more information on Proto3, which Amino is largely compatible with (but not with Proto2).
Due to Amino having significant performance drawbacks, being reflection-based, and not having any meaningful cross-language/client support, Protocol Buffers, specifically gogoprotobuf, is being used in place of Amino. Note, this process of using Protocol Buffers over Amino is still an ongoing process.
Binary wire encoding of types in the Cosmos SDK can be broken down into two main categories, client encoding and store encoding. Client encoding mainly revolves around transaction processing and signing, whereas store encoding revolves around types used in state-machine transitions and what is ultimately stored in the Merkle tree.
For store encoding, protobuf definitions can exist for any type and will typically have an Amino-based "intermediary" type. Specifically, the protobuf-based type definition is used for serialization and persistence, whereas the Amino-based type is used for business logic in the state-machine where they may convert back-n-forth. Note, the Amino-based types may slowly be phased-out in the future, so developers should take note to use the protobuf message definitions where possible.
In the codec
package, there exists two core interfaces, BinaryCodec
and JSONCodec
,
where the former encapsulates the current Amino interface except it operates on
types implementing the latter instead of generic interface{}
types.
The ProtoCodec
, where both binary and JSON serialization is handled
via Protobuf. This means that modules may use Protobuf encoding, but the types must
implement ProtoMarshaler
. If modules wish to avoid implementing this interface
for their types, this is autogenerated via buf
Modules are recommended to use collections for handling encoding and decoding of state. Usage of collections handles marshal and unmarshal for you. By default protobuf is used but other encodings can be used if preferred.
Gogoproto
Modules are encouraged to utilize Protobuf encoding for their respective types. In the Cosmos SDK, we use the Gogoproto specific implementation of the Protobuf spec that offers speed and DX improvements compared to the official Google protobuf implementation.
Guidelines for protobuf message definitions
In addition to following official Protocol Buffer guidelines, we recommend using these annotations in .proto files when dealing with interfaces:
- use
cosmos_proto.accepts_interface
to annotateAny
fields that accept interfaces- pass the same fully qualified name as
protoName
toInterfaceRegistry.RegisterInterface
- example:
(cosmos_proto.accepts_interface) = "cosmos.gov.v1beta1.Content"
(and not justContent
)
- pass the same fully qualified name as
- annotate interface implementations with
cosmos_proto.implements_interface
- pass the same fully qualified name as
protoName
toInterfaceRegistry.RegisterInterface
- example:
(cosmos_proto.implements_interface) = "cosmos.authz.v1beta1.Authorization"
(and not justAuthorization
)
- pass the same fully qualified name as
Code generators can then match the accepts_interface
and implements_interface
annotations to know whether some Protobuf messages are allowed to be packed in a given Any
field or not.
Transaction Encoding
Another important use of Protobuf is the encoding and decoding of transactions. Transactions are defined by the application or the Cosmos SDK but are then passed to the underlying consensus engine to be relayed to other peers. Since the underlying consensus engine is agnostic to the application, the consensus engine accepts only transactions in the form of raw bytes.
- The
TxEncoder
object performs the encoding. - The
TxDecoder
object performs the decoding.
loading...
A standard implementation of both these objects can be found in the auth/tx
module:
https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/x/auth/tx/decoder.go
loading...
See ADR-020 for details of how a transaction is encoded.
Interface Encoding and Usage of Any
The Protobuf DSL is strongly typed, which can make inserting variable-typed fields difficult. Imagine we want to create a Profile
protobuf message that serves as a wrapper over an account:
message Profile {
// account is the account associated to a profile.
cosmos.auth.v1beta1.BaseAccount account = 1;
// bio is a short description of the account.
string bio = 4;
}
In this Profile
example, we hardcoded account
as a BaseAccount
. However, there are several other types of user accounts related to vesting, such as BaseVestingAccount
or ContinuousVestingAccount
. All of these accounts are different, but they all implement the AccountI
interface. How would you create a Profile
that allows all these types of accounts with an account
field that accepts an AccountI
interface?
loading...
In ADR-019, it has been decided to use Any
s to encode interfaces in protobuf. An Any
contains an arbitrary serialized message as bytes, along with a URL that acts as a globally unique identifier for and resolves to that message's type. This strategy allows us to pack arbitrary Go types inside protobuf messages. Our new Profile
then looks like:
message Profile {
// account is the account associated to a profile.
google.protobuf.Any account = 1 [
(cosmos_proto.accepts_interface) = "cosmos.auth.v1beta1.AccountI"; // Asserts that this field only accepts Go types implementing `AccountI`. It is purely informational for now.
];
// bio is a short description of the account.
string bio = 4;
}
To add an account inside a profile, we need to "pack" it inside an Any
first, using codectypes.NewAnyWithValue
:
var myAccount AccountI
myAccount = ... // Can be a BaseAccount, a ContinuousVestingAccount or any struct implementing `AccountI`
// Pack the account into an Any
accAny, err := codectypes.NewAnyWithValue(myAccount)
if err != nil {
return nil, err
}
// Create a new Profile with the any.
profile := Profile {
Account: accAny,
Bio: "some bio",
}
// We can then marshal the profile as usual.
bz, err := cdc.Marshal(profile)
jsonBz, err := cdc.MarshalJSON(profile)
To summarize, to encode an interface, you must 1/ pack the interface into an Any
and 2/ marshal the Any
. For convenience, the Cosmos SDK provides a MarshalInterface
method to bundle these two steps. Have a look at a real-life example in the x/auth module.
The reverse operation of retrieving the concrete Go type from inside an Any
, called "unpacking", is done with the GetCachedValue()
on Any
.
profileBz := ... // The proto-encoded bytes of a Profile, e.g. retrieved through gRPC.
var myProfile Profile
// Unmarshal the bytes into the myProfile struct.
err := cdc.Unmarshal(profilebz, &myProfile)
// Let's see the types of the Account field.
fmt.Printf("%T\n", myProfile.Account) // Prints "Any"
fmt.Printf("%T\n", myProfile.Account.GetCachedValue()) // Prints "BaseAccount", "ContinuousVestingAccount" or whatever was initially packed in the Any.
// Get the address of the account.
accAddr := myProfile.Account.GetCachedValue().(AccountI).GetAddress()
It is important to note that for GetCachedValue()
to work, Profile
(and any other structs embedding Profile
) must implement the UnpackInterfaces
method:
func (p *Profile) UnpackInterfaces(unpacker codectypes.AnyUnpacker) error {
if p.Account != nil {
var account AccountI
return unpacker.UnpackAny(p.Account, &account)
}
return nil
}
The UnpackInterfaces
gets called recursively on all structs implementing this method, to allow all Any
s to have their GetCachedValue()
correctly populated.
For more information about interface encoding, and especially on UnpackInterfaces
and how the Any
's type_url
gets resolved using the InterfaceRegistry
, please refer to ADR-019.
Any
Encoding in the Cosmos SDK
The above Profile
example is a fictive example used for educational purposes. In the Cosmos SDK, we use Any
encoding in several places (non-exhaustive list):
- the
cryptotypes.PubKey
interface for encoding different types of public keys, - the
sdk.Msg
interface for encoding differentMsg
s in a transaction, - the
AccountI
interface for encoding different types of accounts (similar to the above example) in the x/auth query responses, - the
EvidenceI
interface for encoding different types of evidences in the x/evidence module, - the
AuthorizationI
interface for encoding different types of x/authz authorizations, - the
Validator
struct that contains information about a validator.
A real-life example of encoding the pubkey as Any
inside the Validator struct in x/staking is shown in the following example:
loading...
Any
's TypeURL
When packing a protobuf message inside an Any
, the message's type is uniquely defined by its type URL, which is the message's fully qualified name prefixed by a /
(slash) character. In some implementations of Any
, like the gogoproto one, there's generally a resolvable prefix, e.g. type.googleapis.com
. However, in the Cosmos SDK, we made the decision to not include such prefix, to have shorter type URLs. The Cosmos SDK's own Any
implementation can be found in github.com/cosmos/cosmos-sdk/codec/types
.
The Cosmos SDK is also switching away from gogoproto to the official google.golang.org/protobuf
(known as the Protobuf API v2). Its default Any
implementation also contains the type.googleapis.com
prefix. To maintain compatibility with the SDK, the following methods from "google.golang.org/protobuf/types/known/anypb"
should not be used:
anypb.New
anypb.MarshalFrom
anypb.Any#MarshalFrom
Instead, the Cosmos SDK provides helper functions in "github.com/cosmos/cosmos-proto/anyutil"
, which create an official anypb.Any
without inserting the prefixes:
anyutil.New
anyutil.MarshalFrom
For example, to pack a sdk.Msg
called internalMsg
, use:
import (
- "google.golang.org/protobuf/types/known/anypb"
+ "github.com/cosmos/cosmos-proto/anyutil"
)
- anyMsg, err := anypb.New(internalMsg.Message().Interface())
+ anyMsg, err := anyutil.New(internalMsg.Message().Interface())
- fmt.Println(anyMsg.TypeURL) // type.googleapis.com/cosmos.bank.v1beta1.MsgSend
+ fmt.Println(anyMsg.TypeURL) // /cosmos.bank.v1beta1.MsgSend
FAQ
How to create modules using protobuf encoding
Defining module types
Protobuf types can be defined to encode:
Naming and conventions
We encourage developers to follow industry guidelines: Protocol Buffers style guide and Buf, see more details in ADR 023