The glTF file format has materials, textures and images. The mesh has an index into a material, the material has as index into a texture - and the texture has an index into an image. The image has the url or offset in the binary file for the image data.
Below shows a simple example of the gltf file contents.
The Tokyo model is a complex model with lots of materials and meshes so makes a good test case, below gives a dump of some of the elements within the file:
After all the data has been loaded in the gltfLoader (parsed everything) - there is a list of images (
parsedData.images
). We then add an extra function called
loadImages(..)
which loads all of the images.
The
loadImages(..)
handles both binary and ascii version of the file - so if the data is ascii the images are either inlined using base64 or are links to external images (usually in the same path at the gltf file). While the binary versions (in the glb file) have the images within the file - so we use the binary data and the offset to extract the images.
The images are loaded and decoded and stored in the
parsedData
structure so they can be used later.
// Utility function to load images
async loadImages(parsedData, gltfPath) {
if (!parsedData.images) return;
const imagePromises = parsedData.images.map(async (image, index) => {
// console.log('image:', image, 'index:', index );
if (image.bufferView !== undefined) {
// Handle embedded images in GLB
const bufferView = parsedData.bufferViews[image.bufferView];
const buffer = parsedData.buffers[bufferView.buffer];
const byteOffset = bufferView.byteOffset || 0;
const byteLength = bufferView.byteLength;
// Check if buffer.data is correctly populated
console.assert(buffer.data, 'data is not populated correctly');
const imageData = new Uint8Array(buffer.data, byteOffset, byteLength);
const blob = new Blob([imageData], { type: image.mimeType });
const img = new Image();
img.src = URL.createObjectURL(blob);
await img.decode();
parsedData.images[index].imageObject = img;
// debug show the image (thumb) - check it's decoded correctly
document.body.appendChild( img );
img.style.width = '64px';
}
else
{
// Fetch can be used for both the inline and external file -
// just got to fix up the url so it's correct
let imageUri = image.uri;
console.log('imageUri:', imageUri );
if ( gltfPath && !imageUri.startsWith('data:') && !imageUri.startsWith('http')) {
imageUri = gltfPath + image.uri;
}
const imageResponse = await fetch(imageUri);
const imageBlob = await imageResponse.blob();
const img = new Image();
img.src = URL.createObjectURL(imageBlob);
try {
await img.decode();
}
catch ( error )
{
console.log('either cannot load image or it cannot be decoded:', image.uri );
console.log('cannot decode image..', image.uri);
}
parsedData.images[index].imageObject = img;
// debug show the image as small thumb for debugging
document.body.appendChild( img );
img.style.width = '64px';
}
});
await Promise.all(imagePromises); // Wait for all images to be loaded
}
Loading the Images into the GPU
Once we have the data ready we can setup or renderer to use it. The WebGPU shader lets use create an Array of Textures so we can pass all the textures through to the shader and access them using an index.
This is relatively straightforward as shown below - the only aspect to be careful about - if you use an array of textures - all your textures have to be the same size! You also have to make sure that your images are the correct size - otherwise, when you map your textures onto the mesh you'll end up with the textures not looking correct.
You can see an example of incorrect image scaling - on the left is if we do not 'resizeImage(..)' to the same size as the image array size - while the image on the right uses the corrected image size for the array.
<?php
// Helper function for resizing the images
const resizeImage = async (originalImage, newWidth, newHeight) => {
// Create a canvas element
const canvas = document.createElement('canvas');
canvas.width = newWidth;
canvas.height = newHeight;
// Get the 2D drawing context
const ctx = canvas.getContext('2d');
// Draw the original image onto the canvas with new dimensions
ctx.drawImage(originalImage, 0, 0, newWidth, newHeight);
// Create a new Image object
const resizedImage = new Image();
// Set the source of the new Image to the data URL of the canvas
resizedImage.src = canvas.toDataURL();
await resizedImage.decode();
return resizedImage;
}
// create the texture of ararys which we pass to the shader through the bind group
let layerCount = parsedData.images ? parsedData.images.length : 1;
const textureWidth = 512;
const textureHeight = 512;
const textureArray = device.createTexture({
size: [textureWidth, textureHeight, layerCount],
format: 'rgba8unorm',
usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT,
});
// process the images and copy them across to the texture array (if we have images)
if ( parsedData.images )
{
layerCount = parsedData.images;
for (let i=0; i<parsedData.images.length; i++)
{
let img = parsedData.images[i].imageObject;
let img2 = await resizeImage( img, 512, 512 );
const imageBitmap = await createImageBitmap(img2);
const bitmapWidth = imageBitmap.width;
const bitmapHeight = imageBitmap.height;
console.assert( bitmapWidth==textureWidth && bitmapHeight==textureHeight );
device.queue.copyExternalImageToTexture(
{ source: imageBitmap },
{ texture: textureArray, origin: { x: 0, y: 0, z: i } },
{ width: textureWidth, height: textureHeight, depthOrArrayLayers: 1 }
);
}
}
// Create sampler - used for sampling the texture in the fragment shader
const sampler = device.createSampler({ magFilter: 'linear', minFilter: 'linear' });
Managing Materials, Textures and Images
Once all the data is parsed and ready - you can create a material buffer on the GPU which can store all the material data. Each mesh has a material index - so you can lookup the correct material in the fragment shader as needed.
For the example below, we define a maximum nunber of materials and create an uniform array - for testing the materials are just an array of vec4s - but this can be extended as needed for more information to be passed to the shader.
The material index is stored in the w component of the vertex - as by default each vertex is a vec3 - but we use vec4s and store the material index in the last component. Pass this value through to the fragment shader so we can lookup the material in the array of uniform materials.
The Tokyo scene with basic decal shading is shown below (including the animation). Padding more material information to the shader you can add further lighting calculations to increase the realism of the scene.
An output with the animation for the Tokyo scene.
Things to Try
• Add in additional materials (pass them to the fragment shader and implement the calculation)
• Accesss additional texture data (e.g., specular and normals)
• Modify the camera so you can fly around the Tokyo scene
• Put the camera in the front of the tram car and update it so it moves with the tram (feels like you're inside at the front of the tram car)
• Add a skymap (cubemap) so you can add atmospheric feeling (sun/sky moving/reflections)
• Add in some shadows (simple shadow map)
Resources and Links
• Tokyo Model Dumped with Textures/Materials (LINK)
Visitor:
Copyright (c) 2002-2026 xbdev.net - All rights reserved.
Designated articles, tutorials and software are the property of their respective owners.
