ADR 031: Protobuf Msg Services

Changelog

• 2020-10-05: Initial Draft
• 2021-04-21: Remove ServiceMsgs to follow Protobuf Any's spec, see #9063 (opens new window).

Status

Accepted

Abstract

We want to leverage protobuf service definitions for defining Msgs which will give us significant developer UX improvements in terms of the code that is generated and the fact that return types will now be well defined.

Context

Currently Msg handlers in the Cosmos SDK do have return values that are placed in the data field of the response. These return values, however, are not specified anywhere except in the golang handler code.

In early conversations it was proposed (opens new window) that Msg return types be captured using a protobuf extension field, ex:

Copy package cosmos.gov; message MsgSubmitProposal option (cosmos_proto.msg_return) = “uint64”; string delegator_address = 1; string validator_address = 2; repeated sdk.Coin amount = 3; }

This was never adopted, however.

Having a well-specified return value for Msgs would improve client UX. For instance, in x/gov, MsgSubmitProposal returns the proposal ID as a big-endian uint64. This isn’t really documented anywhere and clients would need to know the internals of the Cosmos SDK to parse that value and return it to users.

Also, there may be cases where we want to use these return values programatically. For instance, https://github.com/cosmos/cosmos-sdk/issues/7093 proposes a method for doing inter-module Ocaps using the Msg router. A well-defined return type would improve the developer UX for this approach.

In addition, handler registration of Msg types tends to add a bit of boilerplate on top of keepers and is usually done through manual type switches. This isn't necessarily bad, but it does add overhead to creating modules.

Decision

We decide to use protobuf service definitions for defining Msgs as well as the code generated by them as a replacement for Msg handlers.

Below we define how this will look for the SubmitProposal message from x/gov module. We start with a Msg service definition:

Copy package cosmos.gov; service Msg { rpc SubmitProposal(MsgSubmitProposal) returns (MsgSubmitProposalResponse); } // Note that for backwards compatibility this uses MsgSubmitProposal as the request // type instead of the more canonical MsgSubmitProposalRequest message MsgSubmitProposal { google.protobuf.Any content = 1; string proposer = 2; } message MsgSubmitProposalResponse { uint64 proposal_id; }

While this is most commonly used for gRPC, overloading protobuf service definitions like this does not violate the intent of the protobuf spec (opens new window) which says:

If you don’t want to use gRPC, it’s also possible to use protocol buffers with your own RPC implementation. With this approach, we would get an auto-generated MsgServer interface:

In addition to clearly specifying return types, this has the benefit of generating client and server code. On the server side, this is almost like an automatically generated keeper method and could maybe be used intead of keepers eventually (see #7093 (opens new window)):

Copy package gov type MsgServer interface { SubmitProposal(context.Context, *MsgSubmitProposal) (*MsgSubmitProposalResponse, error) }

On the client side, developers could take advantage of this by creating RPC implementations that encapsulate transaction logic. Protobuf libraries that use asynchronous callbacks, like protobuf.js (opens new window) could use this to register callbacks for specific messages even for transactions that include multiple Msgs.

Each Msg service method should have exactly one request parameter: its corresponding Msg type. For example, the Msg service method /cosmos.gov.v1beta1.Msg/SubmitProposal above has exactly one request parameter, namely the Msg type /cosmos.gov.v1beta1.MsgSubmitProposal. It is important the reader understands clearly the nomenclature difference between a Msg service (a Protobuf service) and a Msg type (a Protobuf message), and the differences in their fully-qualified name.

This convention has been decided over the more canonical Msg...Request names mainly for backwards compatibility, but also for better readability in TxBody.messages (see Encoding section below): transactions containing /cosmos.gov.MsgSubmitProposal read better than those containing /cosmos.gov.v1beta1.MsgSubmitProposalRequest.

One consequence of this convention is that each Msg type can be the request parameter of only one Msg service method. However, we consider this limitation a good practice in explicitness.

Encoding

Encoding of transactions generated with Msg services do not differ from current Protobuf transaction encoding as defined in ADR-020. We are encoding Msg types (which are exactly Msg service methods' request parameters) as Any in Txs which involves packing the binary-encoded Msg with its type URL.

Decoding

Since Msg types are packed into Any, decoding transactions messages are done by unpacking Anys into Msg types. For more information, please refer to ADR-020.

Routing

We propose to add a msg_service_router in BaseApp. This router is a key/value map which maps Msg types' type_urls to their corresponding Msg service method handler. Since there is a 1-to-1 mapping between Msg types and Msg service method, the msg_service_router has exactly one entry per Msg service method.

When a transaction is processed by BaseApp (in CheckTx or in DeliverTx), its TxBody.messages are decoded as Msgs. Each Msg's type_url is matched against an entry in the msg_service_router, and the respective Msg service method handler is called.

For backward compatability, the old handlers are not removed yet. If BaseApp receives a legacy Msg with no correspoding entry in the msg_service_router, it will be routed via its legacy Route() method into the legacy handler.

Module Configuration

In ADR 021, we introduced a method RegisterQueryService to AppModule which allows for modules to register gRPC queriers.

To register Msg services, we attempt a more extensible approach by converting RegisterQueryService to a more generic RegisterServices method:

Copy type AppModule interface { RegisterServices(Configurator) ... } type Configurator interface { QueryServer() grpc.Server MsgServer() grpc.Server } // example module: func (am AppModule) RegisterServices(cfg Configurator) { types.RegisterQueryServer(cfg.QueryServer(), keeper) types.RegisterMsgServer(cfg.MsgServer(), keeper) }

The RegisterServices method and the Configurator interface are intended to evolve to satisfy the use cases discussed in #7093 (opens new window) and #7122 (opens new window).

When Msg services are registered, the framework should verify that all Msg types implement the sdk.Msg interface and throw an error during initialization rather than later when transactions are processed.

Msg Service Implementation

Just like query services, Msg service methods can retrieve the sdk.Context from the context.Context parameter method using the sdk.UnwrapSDKContext method:

Copy package gov func (k Keeper) SubmitProposal(goCtx context.Context, params *types.MsgSubmitProposal) (*MsgSubmitProposalResponse, error) { ctx := sdk.UnwrapSDKContext(goCtx) ... }

The sdk.Context should have an EventManager already attached by BaseApp's msg_service_router.

Separate handler definition is no longer needed with this approach.

Consequences

This design changes how a module functionality is exposed and accessed. It deprecates the existing Handler interface and AppModule.Route in favor of Protocol Buffer Services (opens new window) and Service Routing described above. This dramatically simplifies the code. We don't need to create handlers and keepers any more. Use of Protocol Buffer auto-generated clients clearly separates the communication interfaces between the module and a modules user. The control logic (aka handlers and keepers) is not exposed any more. A module interface can be seen as a black box accessible through a client API. It's worth to note that the client interfaces are also generated by Protocol Buffers.

This also allows us to change how we perform functional tests. Instead of mocking AppModules and Router, we will mock a client (server will stay hidden). More specifically: we will never mock moduleA.MsgServer in moduleB, but rather moduleA.MsgClient. One can think about it as working with external services (eg DBs, or online servers...). We assume that the transmission between clients and servers is correctly handled by generated Protocol Buffers.

Finally, closing a module to client API opens desirable OCAP patterns discussed in ADR-033. Since server implementation and interface is hidden, nobody can hold "keepers"/servers and will be forced to relay on the client interface, which will drive developers for correct encapsulation and software engineering patterns.

Pros

• communicates return type clearly
• manual handler registration and return type marshaling is no longer needed, just implement the interface and register it
• communication interface is automatically generated, the developer can now focus only on the state transition methods - this would improve the UX of #7093 (opens new window) approach (1) if we chose to adopt that
• generated client code could be useful for clients and tests
• dramatically reduces and simplifies the code

Cons

• using service definitions outside the context of gRPC could be confusing (but doesn’t violate the proto3 spec)

References

• Initial Github Issue #7122 (opens new window)
• proto 3 Language Guide: Defining Services (opens new window)
• Initial pre-Any Msg designs (opens new window)
• ADR 020
• ADR 021