Analyzing and processing lager datasets
When it comes to image processing, a simple workaround for memory availability is what is sometimes referred to as a divide-and-conquer approach or blocking. When using blocking, the software loads blocks of image data into the memory, then processes that block of data, and finally writes the results to a temporary document on the hard disk before loading the next block for processing. This very efficient method of processing does impose some constraints.
First, we need to have enough spare hard disk space to hold the processed images. However, since disk storage is generally significantly cheaper than RAM and there is usually a lot more of it available this is actually an asset of this approach. Our article outlining the System requirements has further details on this point.
Second, a lot of image processing algorithms will be affected by the boundary pixels, meaning that we need to load a large enough margin on the side of the block we are interested in to allow for seamless stitching of the processed blocks.
Third, some image processing algorithms require the entire image to be held in memory. Therefore, as a result of our decision to make sure the software works with any images of any size on any computer we have specifically decided not to use these algorithms.
This topic is covered in more details in this article that explains why it may look like arivis is not using the full system resources.