Time lapse analysis

Object tracking is the process of identifying objects and features and measuring changes in these objects over time. It is different from looking at a simple before/after, 2-timepoint dataset and measuring macro changes. Instead, tracking focuses on dynamic changes of specific objects through a series of consecutive time points to better understand dynamic processes affecting the changes that can be seen from start to finish.

Tracking systems take a series of images taken over multiple time points, and then try to identify the same specific objects or features from one time-point to the next so that we can measure and monitor specific changes.

Examples of tracking applications include:

• Calcium ratio experiments - We monitor fluorescence intensity changes between cells
• Contact analysis - We track specific cells and try to identify the frequency and duration of contact events with other structures in the image, like blood vessels or other cell types
• Lineage tracking - Monitor where embryonic cells are, how often they divide and where the children of these divisions go
• Wound healing - Measure the speed at which cells migrate to close a gap between to separate cell tissues

And many others besides. But what all these applications have in common is that we are monitoring how a specific feature (wound area, velocity, intensity, etc) changes over a time series. 

The process of track creation can generally be considered as two separate processes:

• Identifying and marking specific objects or features
• Correlating these objects over the course of the time series to create a special group we call a track.

Both of these tasks can be carried out in a variety of ways depending on the specific application.

For example, object recognition can be done through automatic segmentation of an image, or it could be done by creating a simple region of interest and duplicating that region over all the available time points. Tracking can be done manually by a user making interpretations of the image data, or algorithmically by identifying segmented objects from one time-point to the next.

With this in mind, the tracking accuracy is highly dependent both on the ability to recognize the same object accurately from one time-point to the next, and the ability to recognize the objects in each time point individually in the first place. Both of these are highly dependent on the quality of the image data and the sampling frequency.

Along with image quality being a very significant factor in segmentation, sampling resolution is also critical for correct tracking. If the movement is too great compared to the typical object separation, correct identification of movement can be challenging or even impossible.

If, however, we can increase the acquisition frequency so that the movement from one time-point to the next the confidence in the correct identification of tracks improves significantly.

As a rule of thumb, it is generally preferable to take images frequently enough that the typical movement of objects from one time-point to the next is no more than 20% of the typical distance between neighboring objects.

Similarly, in cases where we are looking to monitor intensity changes, it's important to take images frequently enough to measure those changes accurately. If those changes are rhythmical or affected by rhythms in the sample (e.g. heartbeats), the acquisition frequency for each time-point should be at least 4 times higher than the frequency of changes in the sample. 

All this taken together means that in many cases tracking will not be possible and looking at macro before/after changes may be the most suitable type of analysis. Also, in many cases, the sampling frequency needs, coupled with the exposure time needed to capture images of sufficient quality may restrict the acquisitions to single planes thereby limiting the ability to measure changes in 4D.

Having acquired images for the purpose of tracking, the first task will be to establish an effective segmentation strategy. Several factors can complicate good segmentation throughout the time series. For example:

• The signal or background intensity may change throughout the time series.
• Objects coming into contact with other objects will need to be effectively separate.
• Object morphologies may change significantly throughout the time series resulting in the need for relatively permissive filters. 

All of these factors can, to some extent, be handled by image and object processing algorithms.

Once the segmentation has been optimized, we can add the tracking operation to our pipeline.

The tracking operation offers a range of parameters that must be set according to the needs of the analysis. As stated above, when tracking we identify objects at every time point and then try to establish the connection between these objects. What is allowed or not is depending on the parameters of the operation. Broadly these parameters tell the algorithm:

• How the objects are likely to move - are they likely to move in a more or less straight line or is the movement more chaotic?
• How far they are likely to move - please see above regarding sampling resolution
• How much do we expect the objects to change - do we expect to mostly retain their shape/intensity, or change, are they expected to merge/divide?

Once the tracking parameters have been set as needed the operation can be run and the segmented images will be analysed to create tracks. The outputs are:

• A visual representation of the tracks on the image that allows the users to evaluate visually whether the results are correct
• 
• Entries for each track in the Objects table where the users can review specific features of the tracks, including features of the tracks (displacement, duration, number of segments in the track, etc), and features of the segments in the tracks sorted by the track they belong to.

This enables the user to do a few things.

First, if the tracks appear wrong we can revert the tracking operation, change the parameters so as to try and improve the results, and run the operation again. If no good tracking parameters exist because the image doesn't provide good automatic tracking conditions, manual correction of the tracks is also possible. 

Secondly, the objects table can be configured to display features of the tracks and their segments. Since the tracks are essentially groups of objects it is often best to switch the Objects table display to the Master/Detail view, which shows the tracks in the upper table and the segmented objects in those tracks in the lower table.

Each table can then be configured individually to show pertinent information for both tracks, and the segments in those tracks.

Finally, the results can be exported, either as a pipeline operation, or from the Object window's Im/Export... menu.

The tracking tools in arivis Vision4D are very good, but tracking results are unlikely to be perfect. A range of options are available if the results of the tracking operation are unsatisfactory:

• The first preferred option should be to try and improve either the segment detection parameters or the tracking parameters. Improving the results for the automated tracking reduces the time required to do the analysis and reduces the need for manual interaction.
• If neither the segmentation nor the tracking can be improved any further it may be that improving the acquisition to facilitate the analysis may be the next best option.
• Of course, acquiring new data is often not practical, and in any case, the acquisition results may be the best that are technically achievable, and in that case, manually editing the tracks is also a possibility.

The Track Editor can be found in the Objects menu.

In the Track editor, users can link, split, and merge tracks, as well as assign segments to tracks if they were missed, or manually remove individual segments incorrectly assigned to a track. 

In this example, we have two tracks that appear like they should be connected but are not:

In such cases, the first thing to do is double check that they are indeed tracks that should be connected as it could be that they only appear so due to the current visualisation parameters. Tracks could appear to be connected but shouldn't be because:

• The perspective suggests a continuation but a different angle would reveal otherwise.
• The two track segments may be temporally divergent and playing back through time to reveal the segments might reveal that they are not in fact connected.

Once the connection is confirmed, editing the tracks is as simple as drawing a line between the last time point where the track was correct, and the next time point.

Further details on track editing are available in the help files.

• Optimize your segmentation as much as possible. Tracking is dependent on segmentation results and the better these are, the more accurate the tracking will be.
• Try different parameters to see which work best for your data, especially with regards to the motion types.
• Only use what you need:
• Reduce the search radius to avoid big jumps in the tracks.
• Only use weighing if the segments don't change shape/intensity much over time.
• Don't allow fusions or divisions unless this is necessary.
• Avoid manual editing tracks to reduce the time required to do the analysis and reduce user biases.
• Consider using VR for manual track editing and creation is automated 4D tracking doesn't work for your images.

1. Select Open... from the file menu.
   
2. Select the dataset from the file browser.
   The dataset is a multi dimensional, discrete, representation of your real sample volume. It can be structured as a Z series of planes (Optical sectioning) of multiple channels (dyes) in a temporal sequence of time points (located in several spatial positions).​
   Usually, the dataset shows a single experimental situation (a complete experiment can be composed by several datasets). The datasets are available as graphic files saved in plenty of file formats (standard formats as well as proprietary formats)​.
   Note: The dataset is visualized according to the current rendering setting parameters. Refer to the arivis Pro Help for more details.
   

1. Set the active time point as time #1.
   The time points can be manually selected in the following ways:
   Using the scroll bar in the Navigator panel.
   
   Using the time tools on the main icon bar (must be set in the preferences).
   
   
   Using short keys. By default, the time changes are assigned to the CTRL + B / Ctrl + F keys (the keys assignement can be changed in the preferences).
   Note: The short key method is the suggested one.
2. Set the active cursor mode as following (mandatory):
   
3. Switch back to the white arrow if you need to interact with the volume.
   
   Note: The volume visualization can be disabled if necessary (only in 4D view mode).
4. Click on the segment you want to identify to be tracked.
5. The segment will be highlighted either on both the image and on the object table.
   
5. Go to the next time point (Ctrl + F).
6. Click on the same segment to identify it.
7. Keep the Shift key pressed while clicking on the object.
8. The segment will be highlighted either on the image and in the object table.
   
8. Repeat the selection until the segment path is completed.
   Note: From the second time point up to the end of the segment selection (over time), the Shift key MUST be kept pressed while clicking on the object.
9. On the object table, right-click on one of the selected segments (grayed) to identify it.
   
10. From the pop up menu, select Add Tags...
    
    
11. Type in the new TAG name (e.g. Track#1). Press Add Tag.
    
    Note: Don‘t click anywhere else. This will deselect all the segments.
12. The new TAG is shown inside the object table. It includes all the selected segments (please check).
    

Multiple segments can be grouped in different tags following the described procedure.

1. To modify the tracks, open the Track Editor.
   
   
2. The tracks are listed on the left panel. The tracked objects belonging to the specific track are shown on the right panel.
2. Click on the track.
   
3. Select the time point and the segment to be edited. The related frame is shown in the viewing area. The track and the segment are highlighted.
   
4. Right-click on the selected time point.
5. A pop up menu is displayed.
   
   The track can be splitted and merged with another one. The segments in the track path can also be deleted.

The key to identifying interactions is to tag subjects based on their proximity to or overlap with reference objects. To enable this step both types of objects must first be segmented and classified. Use any segmentation tool that is appropriate and use filters if necessary to identify only those segmented objects that are likely subject by, for example, not tagging objects outside of specific volume ranges or other features. 

Having tagged contact events we can then go on to tracking the objects.

Here the tracking operator is configured as follows:

1. The input for the tracking operation is all the cells that can make contact, whether they do or not is not important at this stage. Use the menu in the operation title bar of the operation to select the input.
2. The track colouring is fixed This is done so that we can more easily visualise contact events in the viewer later by combining the track colour with the object colour.
3. The tracking parameters have been set according to the specific dataset. Check the knowledge base article on tracking for more information.
4. The option to colour segment by tracks has been disabled so that the object colours are maintained.

Having tracked the objects, there are a few things that can be done to help interpret the data and extract the required information. Most of it is done from the Objects table.