So what can we do to optimize the processing of such potentially large datasets?
Generally, in computing, there are a few specific bottlenecks that limit the ability to view and process data. These are:
- The number of display pixels - Most computer systems only have between 2 and 8 million pixels of screen real estate. A larger display with more pixels will also typically require more memory and a better GPU.
- Amount of memory available to the system - Most computers come with 8-16GB of memory as standard. High-end workstations can be configured with up to 2-4TB of memory, but these workstations will typically cost 10s of thousands of dollars, of which the memory could be as much as 2/3rds of the cost.
- Central Processing Unit capabilities - Processing power is generally limited by several factors, from clock speed to core architecture. A common strategy to boost computing power is to use parallelization to split the processing across multiple cores. Not all tasks can be parallelized.
- Graphics Processing Unit capabilities - GPUs are essentially small computing units inside your computer dedicated to the task of displaying information in 3D. They also have limitations with regards to the number and speed of the cores that can be built into a chip, and the amount of video memory available.
- Read/write speeds and storage capacity - As we'll see below, we can use temporary documents on the hard disk to work around memory limitations, but hard disks read/write speed can become important, especially when considering the cost/speed/capacity compromise for differing storage technologies. Also, many file formats don't readily allow a program to load only part of a file.
Visualizing larger datasets
First, in terms of pure visualization, as mentioned above most computer displays only have 2 to 8 million pixels of display real estate. When a TB image is loaded, it is almost impossible to physically display every pixel from the dataset on the screen at one time. Therefore, a system that spends minutes or hours loading the entire image in memory to only display a small fraction of these pixels at any one time is generally wasteful. Instead, arivis built a system that only loads the pixels that we need, at the resolution we need, as and when we need them.
Typically this is done by creating a so-called pyramidal file structure, where the image is stored with multiple levels of redundancy with regard to zoom levels, so that when you load the image at the lowest zoom level you only load the lowest resolution layer, and as the user zooms in we remove that from the memory and load the next layer with a higher resolution. This is generally very efficient with regards to memory use, but highly inefficient with regards to data storage as the pyramidal file can end up being 1.5x time larger than the raw data just to give this ability to load the resolution as needed. Instead, arivis uses a redundancy-free file structure where the resolution is calculated as needed, meaning that even without compression the converted SIS file will typically be as little as a few 100MB larger than the raw image data. On top of this, we can also apply lossless compression, of the same type as is used in standard compressed file formats like ZIP, to get between 20-80% compression ratios depending on the image data without any loss of information that is critical for scientific image analysis.
We cover 3D visualization of large datasets in more details here, but in short, here is a summary of the visualization concerns in 3D.
When rendering a dataset in 3D, similar constraints exist with regards to the number of pixels that can be displayed on the screen, but also the rendering capabilities of the system are also going to be limited by the graphics capabilities of the GPU and the available video memory. So, likewise, when rendering in 3D arivis will first load in memory only as much of the image data as can be readily handled by the GPU to enable a smooth user experience. This leads to a small, but usually acceptable, loss in resolution for the purpose of interactive visualization and navigation. arivis Vision4D also gives the possibility to render at the highest level of resolution as and when needed and when user interactivity is no longer required.
Analyzing and processing lager datasets
When it comes to image processing, a simple workaround for memory availability is what is sometimes referred to as a divide-and-conquer approach or blocking. When using blocking, the software loads blocks of image data into the memory, then processes that block of data, and finally writes the results to a temporary document on the hard disk before loading the next block for processing. This very efficient method of processing does impose some constraints.
First, we need to have enough spare hard disk space to hold the processed images. However, since disk storage is generally significantly cheaper than RAM and there is usually a lot more of it available this is actually an asset of this approach. Our article outlining the System requirements has further details on this point.
Second, a lot of image processing algorithms will be affected by the boundary pixels, meaning that we need to load a large enough margin on the side of the block we are interested in to allow for seamless stitching of the processed blocks.
Third, some image processing algorithms require the entire image to be held in memory. Therefore, as a result of our decision to make sure the software works with any images of any size on any computer we have specifically decided not to use these algorithms.
This topic is covered in more details in this article that explains why it may look like arivis is not using the full system resources.
The arivis file format
All of the approaches mentioned above require a method of file access that permits the software to readily load arbitrary blocks of data from the file which not all files allow. To this end, arivis developed what we call arivis ImageCore. ImageCore is a combination of a file format that enables arbitrary access to any part of the file, and the libraries implemented in the software that allows us to make use of this feature. Because this is central to the way arivis solutions operate it also means that we do not *open* files such as TIFF, PNG, or any of the common microscopy imaging formats (CZI, LIF, OIF, ND2, etc), but instead we import the data contained in those files into a new file capable of storing all the information and make it accessible in a way that makes it readily accessible to our software.
The files that arivis creates use the ".SIS" extension. SIS files are multidimensional-redundancy free-pyramidal file structures with lossless compression support. This means that:
- We can load the resolution that we need to display the images as and when we need them without needing huge amounts of RAM
- We do so without adding a lot of redundant data to handle the different zoom levels, keeping the imported file size relatively small
- We can handle images of virtually infinite sizes in a range of dimensions. SIS files support virtually limitless numbers of channels, planes, time points, and tiles of any width and height. The only real limit is the size of the storage in your computer. SIS files are technically 7-dimensional, where each pixel can be anything from 8-bit to 32-bit floating-point, and each pixel belongs to a specific X, Y, and Z location in a specific channel, time point, and image set.
- With lossless compression, we can further reduce the size of the imported file by 20-80% depending on the image data.