Announcing Hydra

by joystream

The Joystream development team is very happy to announce Hydra, an internal tool we have built and now wish to make available to the entire Substrate and Polkadot ecosystem.

The Problem

Anyone who has tried to build a non-trivial user experience powered by a Substrate blockchain has undoubtedly run into a number of painful obstacles

  1. Slow Fetching: Doing multiple read operations across multiple block heights is often required, and the pure latency of fetching the data is very bad. Some reads may depend on the result of prior ones, which results in sequential reads, which is even slower.
  2. Slow Processing: The client has to do all the work of filtering, sorting, paginating or otherwise computing on the resulting data, and this takes time, resulting in a slow client experience and CPU load on the user device.
  3. Complex Client: The fetching and processing logic is reproduced in all apps, and adds to the complexity of application development.
  4. No Search: Your chain may have transactional or state commitments to free-text data, but doing real full-text search client-side is not an option.
  5. Over-Fetching: You cannot select the data you want, you have to read full state objects as they are, wasting both connection and full node capacity.
  6. Costly Archival Nodes: User requests which involve multiple read operations. To fetch a consistent dataset, you will often have to read at some specific block height, as always reading at the tip can break atomicity. This means such read requests will depend on the full node being archival. As your user base scales, that will become a very costly infrastructure.

The Solution

Hydra is inspired by TheGraph -- an Ethereum-focused project allowing anyone to deploy a query node exposing historical smart contract data. DApp developers no longer need to build a data pipeline ingesting and indexing historical smart contract data, focusing on the core features of the product.

Hydra is a query node for Substrate-based blockchains. A query node ingests historical events and extrinsics and provides rich, domain-specific, and highly customizable access to the underlying data, far beyond the scope of direct RPC calls.

For example, expired Kusama Treasury spending proposals are pruned from the state of the Kusama blockchain, so querying, say, one-year-old proposals becomes problematic. Indeed, one has to track the evolution of the state by sequentially applying the Treasury events and extrinsics in each historical block.

That's where Hydra gets you covered. Define your data model and the Hydra indexer will get it in sync with the chain. On top of that, you get a batteries-included GraphQL server with comprehensive filtering, pagination, and even full-text search capabilities.

The whole set-up requires just a declarative data model definition in the form of a GraphQL schema file, and a set of "mappings", which tell the indexer how the historical events and extrinsics should be handled. The rest is done by Hydra, which outputs deployable Docker images for the indexer and the GraphQL server. A minimal, but fully functional GraphQL API for historical Kusama proposals can be built and deployed in just a few minutes.

To get a feeling of how a deployed query node looks like, go to the playground and e.g. query rejected proposals with bond value greater than 800000000000000:

query {
proposals(where: { bond_gt: "10000", value_gt: "800000000000000", status_eq: REJECTED }, limit: 5) {
    proposer
    version
    value
    status
    beneficiary
    bond
  }
}

Demo

Here is a nice introductory demo video:

Roadmap

The following are the planned next steps for the framework.

  • Add support for all Substrate FRAME pallet (help us!).
  • Significantly improve dev UX by enforcing type safety and further decoupling indexing and the GraphQL API.
  • Significantly improve performance and synchronization times.

Resources

Check out:

Call to Action!

  1. If you are a developer in this ecosystem, help us get support for all FRAME pallets (we all need them), and generally improve the framework.
  2. Vote for Hydra in the community choice category of the Hackusama Hackathon.

Disclaimer

All forward looking statements, estimates and commitments found in this blog post should be understood to be highly uncertain, not binding and for which no guarantees of accuracy or reliability can be provided. To the fullest extent permitted by law, in no event shall Joystream, Jsgenesis or our affiliates, or any of our directors, employees, contractors, service providers or agents have any liability whatsoever to any person for any direct or indirect loss, liability, cost, claim, expense or damage of any kind, whether in contract or in tort, including negligence, or otherwise, arising out of or related to the use of all or part of this post, or any links to third party websites.