Using arivis Pro

arivis solutions are widely reputed for their ability to handle large image data. At the core of this ability is the way that we store, access and process image data. A more thorough explanation of why we do things this way and the basic concepts behind it can be found here, but for now, it suffices to state that Vision4D requires images to be saved in a specific format to work. 

The first two points to note on this are:

• Import only needs to be done once.
  Importing is the process of copying the image data from whatever format it is saved in, into a new SIS file that Vision4D can open. Once the SIS file is created that file can be used from then on for the rest of the visualization and analysis workflow and the original file can be kept as a backup if needed
• Importing is usually a very straightforward process.
  In most cases, especially those concerning popular imaging file formats such as those of the most common microscope suppliers, importing an image requires very little user interaction beyond simply selecting a save location for the output SIS file. However, arivis does offer advanced import options for cases where the image format is not fully supported.

As mentioned above, importing images is necessary, and usually very straightforward. You can simply drag & drop a file from the file explorer into an open viewer to start the import process. Then, choose a name and save location for the results.

In cases where the file selected is fully supported that is all that is required. The software will automatically read and copy all the image data and metadata. However, it is also possible to import some partially supported files or use more advanced import scenarios when needed.

For example, if the images are stored as multiple TIFF files, the software will be able to read the image information but some of the metadata relating to the dimensionality of the image set may be unrecognizable. In such cases use simple import scenarios, for example, import simple Z-stacks or import time series.

More complex import scenarios are also possible for multidimensional import or imports with stitching of tiles images.

Once the images have been imported into a *.sis file, you can open the file in the same way (drag & drop into an open viewer), or simply double click on the file in the Windows Explorer.

Image segmentation, the process of identifying objects from an image, has been around for as long as digital images. Usually, this process involves creating and using algorithms to classify pixels in an image and then using this classification to identify objects. Various algorithms exist to classify pixels, which often work fast and well but have several limitations regarding the types of images that can be analyzed, and often require a high level of knowledge and skill to apply successfully. whereas humans can usually be trained quickly to recognise objects from images. Deep Learning works by using human knowledge of what constitutes an object to train a model that a computer can use to interpret the image data and produce objects. The exact workings of how a DL network performs the classification are highly complex, but the process of creating a network and applying it to images is accessible to anyone with basic computer skills.

Since arivis Vision4D 3.6 it has been possible to run DL inference for segmentation. This inference can use pre-trained models created with apeer and other platforms that use ONNX models. This only requires that the Vision4D license include the Analysis module.

With Vision4D/arivis Pro 4.1 it is possible to train models directly in the application. This requires a license that includes the AI toolkit and also requires that the arivis Deep Learning package for GPU acceleration be installed.

Once trainings have been created, actually using them in a pipeline is just as easy as using any other pipeline operation, and the results of a DL segmentation can be used in a pipeline in exactly the same way as any other pipeline segments. This includes the ability to:

• Apply object modifications like splitting, dilating, or eroding
• Classify segmented objects based on any of the available object features, and filter out objects that don't meet any given criteria
• Establish Parent/Child relationships or measure distances between objects, including with other objects that were not created through DL
• Export object features and segments masks if needed

The simplest way to use a trained network is to create a new pipeline that uses this network immediately after training. At the bottom of the DL Trainer panel, once the training is complete we can click on Open in pipeline. The software will automatically open the analysis panel, create a new pipeline, add the Deep Learning Segmenter operation to the pipeline, and select the trained model for use with the operation.

With our trained model in the pipeline, we can use this segmentation operation and its output like any other pipeline segmentation operation.

See this other article one using Deep Learning in pipelines for more information on configuring such pipelines to DL models, including custom models created elsewhere, including ZEISS arivis Cloud.

Maximum intensity projections are a great way to render images in 3D as they allow the user to see through the entire depth of an image. However, a MIP can look rather flat unless the volume is rotating or moving making it difficult to get a sense of depth perception from a 2D display. We can use color gradients and apply this to the axis rather than the intensity to better visualize such images.

Once the set has been re-colored we can change the mapping to any axis instead of the intensity from the 4D Opacity Settings panel.

Pipelines are a great tool to extract information from images, and the Objects window can provide a lot of powerful insights through its interactive interface with many display options. However, when dealing with multiple image sets or if statistical analysis tools are required that are not included in Vision4D then exporting the results of the analysis to a spreadsheet can be really useful.

By default, both the Objects table and the analysis panel's Export Object Features operation default to saving the results as MS Excel .XLSX files. 

And while XLSX is a widely supported format, using alternatives like CSV can be preferable in some circumstances.

If the pipeline is used in a batch analysis, the options set in the pipeline remain. Therefore, if a CSV file is the required output in a batch, this should be defined in the pipeline operation, not the Batch Analysis window.

Using CSV export in batch analysis also allows the batch to concatenate analysis results from multiple image sets. To enable this option, start by opening the Batch Analysis window, and after clicking the Next button select the option to "Combine exported CSV files".

Finish by setting up the other export options (output folder and nomenclature) and click Run to start the analysis.

If the pipeline has been configured to export CSV files in arivis Vision4D, it will also export CSV files in VisionHub. To review the results, simply double click on an analysis job and the options to view or download the CSV results will be available on the Results pane on the right of the window.

The image below is a typical example of uneven illumination between the center and the edges of the field-of-view. The same structures will have a different intensity range if located closer to the borders than the center. This makes their segmentation very complex.

The shading effect is much more visible in case of fields stitching. The dark border pattern makes the reconstruction imperfect.

Arivis Pro offers several background subtraction approaches. All of them are available as operators in the pipeline workflow. The background subtraction results can be saved as new SIS file, new Image-Set or as an additional channel in the active Image-Set for display purposes.

Vision4D offers a wide drawing tool collection, comprising the fully manual methods as well as some semi-automatic approaches. The drawing tool collection is available on the top icon panel. It's content varies according to the viewing method setting.

With the 4D viewing method on, both the Magic Wand tool as well as the tool to place simple geometric shapes, markers or spheres on the volume are available. The 2D viewing method allows access to more tools, including the Manual Drawing tool, the Magic Wand tool, and the simple geometric shapes creator tool. In this modality, the drawn objects can be also edited.

The objects created manually are, to all purposes, evaluable segments, exactly as the objects gathered using the automatic approach (Pipeline). All the available measurements can be quantified.

These objects can also be applied as free shape ROI to mask part of the volume or managed to perform evaluations about between structures' interactions and relationships (Objects Compartmentalization, Objects Co-Localization, etc.).

The TAG is an object property assigned to it by the running operator (only the preprocessing operators don't use the TAGs) and used to link all the detected objects to a common collection. By default, the TAG inherits the name from the operator that has generated it. For example, the objects created by BLOB FINDER will have a TAG called “Blob Finder”. The TAG is the Vision4D's way to create a logical and hierarchical relationship between the objects without duplicating them. A single object can belong to many collections. Therefore, it can have numerous TAG's attached to it. The rule is: one object, many TAGs.

When using fluorescence microscopy, especially for in vitro cell cultures it is common to acquire multiple planes to capture the full depth of the sample. In many cases the 3D information is useful and arivis can use it to segment 3D objects and measure volumetric properties. However, in some cases the main advantage of acquiring a stack is to be able to get the whole sample in focus at the optical resolution that allows us to measure the features of interest. In such cases compressing the stack into a single plane can provide all the information that is required while working from much simpler, and smaller datasets. Maximum Intensity Projections (MIPs) are an efficient way to compress such images into a single plane. 

The first point to note is that many acquisition systems are capable of producing these projection images at the time of capture, or at the time of saving, and that if our imaging device is capable of doing this automatically this may be the most efficient way of producing these images for analysis. This is generally the preferred method because:

• It reduces the amount of data being generated or stored in the first place, especially if we don't intend to treat the images as a 3D stack afterwards.
• Generating a MIPs in arivis is not automated, meaning that we have to actively convert each stack into their respective MIPs. This can be especially time consuming for large datasets that contain multiple fields of view.
• The process of generating a MIP in arivis creates a new file or image set within the existing document, leading to some redundancy in storage or the potential need to do a lot of manual tidying up to remove the uncompressed stacks.
• When dealing with large datasets with multiple image sets, if we save the MIPs as new documents it can lead to a large number of documents to be processed, and if we save the MIPs as new image sets it leads to large files with a large number of sets that will need to be actively selected or removed from the selection for batch processing.

Having said all this, arivis Pro can produce Maximum Intensity Projections from the Projection Viewer.

As mentioned in the introduction, it is best, where possible, to generate the MIPs at the point of image capture. 

If we want to use images in a batch analysis it may be preferable to save the MIPs as new documents to facilitate the batch configuration since different pipeline parameters will most likely be required to process MIPs as to process stacks.

For simple snapshots to be added to presentations or publications, using the copying the viewer content to the clipboard, at the resolution currently displayed on the screen, is the fastest way to export the results, and in most cases produces images of an acceptable quality. Creating MIPs as described here is only necessary if we want to have the full native resolution of the dataset and if we want to be able to use it in image analysis pipelines.

For publication it is also worth considering whether a 3D snapshot or video could be a better output.

The video above shows the complete process of creating, modifying and executing a pipeline for tracking on a single dataset. Let's break down the various steps.

The general process of doing image analysis in arivis is fairly simple. We use the Analysis Panel to create Pipelines. Those pipelines are built from individual operations that work with each other to extract the information we need. Pipelines can then easily be re-used with other images as needed, including in batch mode to streamline the process.

Of course, the process we described here is only the basic principle of pipeline:

• Enhance images if needed
• Use images to segment objects
• Use the segmented objects to extract useful information
• Save the results

The full breadth of what can be done in a pipeline cannot be covered here. The inclusion of Machine Learning and Deep Learning makes it possible to segment objects that were previously impossible to segment, and the pipeline tools allow us to extract all sorts of useful information from these segmentation. Please check our pipeline examples, to find out more about the types of information arivis can extract from images, and to learn more about how individual operations work.

Finally, again since our Knowledge Base and sample pipelines couldn't hope to fully cover what can be achieved in a pipeline, don't hesitate to get in touch with your local ZEISS representative.

The Spine Tracer module was introduced in arivis Pro 4.3 as a complement to the Neuron Tracer functionality introduced in arivis Vision4D 4.0. It is designed to identify and quantify neuronal spines. It requires that neurites have already been segmented in the current pipeline as a prerequisite.

The Spine Tracer is then used to detect and quantify dendritic spines in the immediate vicinity of the detected neurites.

As mentioned above, the first prerequisite for spine detection is having already detected neurites. We can use either the Neurite Tracer or Neuron Tracer for this task. The Spine Tracer operation can be added immediately after the Neurite Tracer in the pipeline, or after some filtering operations.

The Spine Tracer can use 3 different methods to detect spines:

• AI-Assisted - This method uses a built in DL model for spine detection. This model automatically identifies spines around the neurites with little to no image pre-processing required and can be used in a majority of cases.
• Use Segments - In this case another pipeline operations has already previously segmented the spines and the Spine Tracer is only responsible for assigning the spines to their respective neurites. The spines can be segmented in any suitable method.
• Use Probability Map - Much like for the Use Segments option, we can use any suitable pipeline operation to isolate and enhance the spines from the rest of the image data, but now the segmentation itself is carried out by the Spine Tracer.

These options make it easy for the majority of users to carry out spine quantification with little additional effort while providing the flexibility to address more challenging images.

In every case below the first step is to select the input tag for the trace objects. Most pipelines of this type will only have one tag for the traces, but it is possible to have several tracing operations in the same pipeline, we therefore need to take care to select the correct tag for the traces at this stage.