New Image Caching Brings Better Performance to the Token Metadata API

We’ve added a new caching layer for token image data in the Token Metadata API. This change brings better performance, better security, and even thumbnail-sized images.

Story details

Type

Product update

Topic(s)

‘

Engineering

’

‘

Product

’

Published

July 2, 2024

Author(s)

Rafael Cardenas

Contents

The Token Metadata API provides data on tokens in the Stacks ecosystem, including token attributes and their corresponding images (perhaps a token logo in the case of a fungible token or an image in the case of an NFT).

In order to get this data, we first pull the metadata from the smart contract that issues the token, and we then store that data in a Postgres database and serve that data via public endpoints.

In a JSON format, the data for a NFT looks something like this:


{
  "version": 1,
  "name": "Belle's Witch 97",
  "collection": "Belle's Witches",
  "description": "The Witches come with all sorts of accessories, hair styles, eyes, lips, clothes and elements with different rarities! There are 3 races: fairy, gnome and human. Which Witch will cast a spell on you?\n\nThis project was the first NFT created by the digital artist Isabelle Parra and was based on a self-portrait",
  "image": "ipfs://QmUUf7WggwHSQ6gGEPpSordi9yyN6hSexSwhbowxRMnWFo/97.png",
  "edition": 97,
  "attributes": [
    {
      "trait_type": "Background",
      "value": "Pink"
    },
    {
      "trait_type": "Race",
      "value": "Gnome Blue Skin"
    },
    {
      "trait_type": "Clothing",
      "value": "Purple Cape"
    },
    {
      "trait_type": "Hair",
      "value": "Blue Straight Hair"
    },
    {
      "trait_type": "Eyes",
      "value": "Yellow Cat Eyes"
    },
    {
      "trait_type": "Lips",
      "value": "Pink Lips"
    },
    {
      "trait_type": "Accessories",
      "value": "Black Glasses"
    },
    {
      "trait_type": "Elemental",
      "value": "Poison Elemental"
    }
  ]
}

As you can see in the code above, you can see the token’s name, the collection it comes from, and a link to the image itself. That link is a problem.

The Problem With Gateways

NFTs usually store their data in IPFS, and in order to access that data, most users go through an ipfs.io gateway (which we use to access the "image" link in the snippet above).

In other words, when users request a token image from our API, we return the link to the image, not the image itself, and users must then fetch the token image via that linked gateway.

Using this gateway comes with a few problems. First of all, it is heavily rate limited. Second, this gateway may not return the image at all: perhaps the gateway (or you) are not near a node that contains the image. Third, it returns only one image file size, which can often be larger than needed, particularly if you’re only looking to use a thumbnail.

All of these issues can translate to delays. By going through this gateway, you may have to wait several seconds for the image to show up, whether by being throttled from the gateway’s rate limit, the size of the image, or something else.

We know this pain firsthand, since we use the Token Metadata API to fetch data for the Stacks Explorer. That’s why we decided to fix it.

A New Image Caching System

We built an internal image caching system to store that token data ourselves. That system begins with this script.

Now, every time we find new images when processing (or re-processing) tokens, we take the images from the original location. For example, in this snippet, you can see how we fetch an image from the IPFS gateway:


fetch(
  IMAGE_URL,
  {
    dispatcher: new Agent({
      headersTimeout: process.env['METADATA_FETCH_TIMEOUT_MS'],
      bodyTimeout: process.env['METADATA_FETCH_TIMEOUT_MS'],
      maxRedirections: process.env['METADATA_FETCH_MAX_REDIRECTIONS'],
      maxResponseSize: process.env['IMAGE_CACHE_MAX_BYTE_SIZE'],
      connect: {
        rejectUnauthorized: false, // Ignore SSL cert errors.
      },
    }),
  },

Then we process that image file using Node.js. Often images are stored as SVGs in IPFS, a file format that has security issues. So in this step, we transform image files into PNGs, and we also resize images, adding a new thumbnail-sized version to our database.


  .then(async response => {
    const imageReadStream = Readable.fromWeb(response.body);
    const passThrough = new PassThrough();
    const fullSizeTransform = sharp().png();
    const thumbnailTransform = sharp()
      .resize({ width: IMAGE_RESIZE_WIDTH, withoutEnlargement: true })
      .png();
    imageReadStream.pipe(passThrough);
    passThrough.pipe(fullSizeTransform);
    passThrough.pipe(thumbnailTransform);

After that is complete, we upload these images to our own Google Cloud Storage bucket that hosts all of these images, which you can see in this snippet:


async function upload(stream, name, authToken) {
  try {
    const response = await request(
      `https://storage.googleapis.com/upload/storage/v1/b/${GCS_BUCKET_NAME}/o?uploadType=media&name=${GCS_OBJECT_NAME_PREFIX}${name}`,
      {
        method: 'POST',
        body: stream,
        headers: { 'Content-Type': 'image/png', Authorization: `Bearer ${authToken}` },
      }
    );
    if (response.statusCode !== 200) throw new Error(`GCS error: ${response.statusCode}`);
    return `${CDN_BASE_PATH}${name}`;
  } catch (error) {
    throw new Error(`Error uploading ${name}: ${error.message}`);
  }
}

Those images, both the full-sized and thumbnail-sized versions, then get added to a <code-rich-text>token metadata api<code-rich-text> folder in our GCS bucket. For clients using the Token Metadata API, you will now see a new JSON response from our endpoints:


{
  "token_uri": "ipfs://QmUpfBNUnVUzwhbahvRTrSPrQhFnBv1VVwe9t6csCPCF53/97.json",
  "metadata": {
    "sip": 16,
    "name": "Belle's Witch 97",
    "description": "The Witches come with all sorts of accessories, hair styles, eyes, lips, clothes and elements with different rarities! There are 3 races: fairy, gnome and human. Which Witch will cast a spell on you?\n\nThis project was the first NFT created by the digital artist Isabelle Parra and was based on a self-portrait",
    "image": "ipfs://QmUUf7WggwHSQ6gGEPpSordi9yyN6hSexSwhbowxRMnWFo/97.png",
    "cached_image": "https://assets.hiro.so/api/mainnet/token-metadata-api/SPJW1XE278YMCEYMXB8ZFGJMH8ZVAAEDP2S2PJYG.belles-witches/97.png",
    "cached_thumbnail_image": "https://assets.hiro.so/api/mainnet/token-metadata-api/SPJW1XE278YMCEYMXB8ZFGJMH8ZVAAEDP2S2PJYG.belles-witches/97-thumb.png",
    "attributes": [
      {
        "trait_type": "Background",
        "display_type": "",
        "value": "Pink"
      },
      {
        "trait_type": "Race",
        "display_type": "",
        "value": "Gnome Blue Skin"
      },
      {
        "trait_type": "Clothing",
        "display_type": "",
        "value": "Purple Cape"
      },
      {
        "trait_type": "Hair",
        "display_type": "",
        "value": "Blue Straight Hair"
      },
      {
        "trait_type": "Eyes",
        "display_type": "",
        "value": "Yellow Cat Eyes"
      },
      {
        "trait_type": "Lips",
        "display_type": "",
        "value": "Pink Lips"
      },
      {
        "trait_type": "Accessories",
        "display_type": "",
        "value": "Black Glasses"
      },
      {
        "trait_type": "Elemental",
        "display_type": "",
        "value": "Poison Elemental"
      }
    ]
  }
}

As you can see in this snippet, both the full-sized and thumbnail-sized images are hosted at the assets.hiro.so URL alongside a link to the IPFS gateway.

This is great for clients like the Stacks Explorer. These clients rely on high availability services, and they can use the thumbnail-sized images to have pages load much faster.

Better Performance for the Token Metadata API

We have been running this processing for all fungible tokens, and now all fungible tokens use this new GCS gateway hosted by Hiro. We are currently rolling out support for all NFT collections, and soon all NFTs on Stacks will also live in our GCS bucket.

Try out the new endpoint responses yourself and get started in our documentation.

Thanks for
subscribing.

Oops! Something went wrong while submitting the form.

New Image Caching Brings Better Performance to the Token Metadata API

The Problem With Gateways

A New Image Caching System

Better Performance for the Token Metadata API

Related stories

New Features in the Stacks Explorer: Advanced Search and Token Holder Data

Release Roundup - July 2024

Introducing the Runes API