Morphological object classification along with instance segmentation

[Aseem139]

Hi Daniel,

I have recently started using BiaPy and find it really convenient to use. Many congratulations on your Nature Methods papers.I am mainly interested in instance segmentation. I have so far tried the nuclei dataset and I am attaching the config file I used and the training results.

Config File:

3d_instance_segmentation.yaml.txt

Training Results:

I have a few questions:

1. The IOU values for the contour channel don't look all that good even though the overall predictions were good. Is this normal ? Should I result the relative weight of the contour channel ?

2. For the segmentation of mitochondria that I am currently working on, I am interested in trying the classification of 3d objects in addition to instance segmentation. I understand how the training dataset must now have 2 channels, one for instance label and other for class labels. Is there any other changes I must make to my config file to tell the model to do classification along with segmentation ?

3. Are there any other changes you'd suggest in my config file that would improve segmentation quality ?

Thanks in advance for the help.

[iarganda]

Hello @Aseem139!

Thanks a lot for your message. We're glad you like BiaPy!

From your plots, it seems your training is highly overfitting. Which dataset are you using? The 3D demo from the StarDist 0.3.0 release?

[Aseem139]

Sorry I uploaded the wrong charts the first time. I seem to have deleted the training charts from the most recent training that worked perfectly. Add the training output instead.

[iarganda]

I see! Those are very good results indeed. Let me answer your questions:

The IOU values for the contour channel don't look all that good even though the overall predictions were good. Is this normal ? Should I result the relative weight of the contour channel ?

This is normal. The IoU of the B channel (binary masks) is high (~0.82 in test), which means the masks of your objects are well identified. The IoU of the C channel (contour maks) is low (~0.10 in test) but this is expected since it measures the overlap between object contours in the prediction and the ground truth, and they are very thin. Nevertheless, those channels are successfully post-processed to obtain individual instance masks (see how high your F1 values are for IoUs 0.3 and 0.5). Summarizing, I would not worry at all about the low metric value of the C channel.

For the segmentation of mitochondria that I am currently working on, I am interested in trying the classification of 3d objects in addition to instance segmentation. I understand how the training dataset must now have 2 channels, one for instance label and other for class labels. Is there any other changes I must make to my config file to tell the model to do classification along with segmentation ?

Maybe @danifranco can tell us more about it, but I believe you only need to do that and also set the variable N_CLASSES (under MODEL) to your number of classes (including the background class, that should be 0) in your yaml file.

Are there any other changes you'd suggest in my config file that would improve segmentation quality ?

Your metrics are quite high already, is there anything you would like to improve from your results? Border quality?

[danifranco]

Hi!

I'm adding some small details on top of @iarganda 's comments. Let me answer you:

The IOU values for the contour channel don't look all that good even though the overall predictions were good. Is this normal ? Should I result the relative weight of the contour channel ?
I would suggest setting LOSS.CLASS_REBALANCE to True so the contour probs are much higher.

For the segmentation of mitochondria that I am currently working on, I am interested in trying the classification of 3d objects in addition to instance segmentation. I understand how the training dataset must now have 2 channels, one for instance label and other for class labels. Is there any other changes I must make to my config file to tell the model to do classification along with segmentation ?

As @iarganda was saying, only MODEL.N_CLASSES needs to be set ;)

Are there any other changes you'd suggest in my config file that would improve segmentation quality ?
@iarganda : Your metrics are quite high already, is there anything you would like to improve from your results? Border quality?

Yeah, the results are quite good already. Made the change I suggested in order to have higher IoU in contous. You can share with us also an image of failure/region where do you want to improve so I can suggest you a few changes.

[Aseem139]

Hi Ignacio and Daniel,

Thank you very much for the response. I tried to segment the mitoEM dataset, again using the same config file. Here are the results:

This is clearly overfitting and can only be fixed with more training data, right ?

Below are the inference results in 2D and in 3D. While mitochondria are more or less accurately segmented with the exception of end slices, there are a great deal of random false positives. Could these be taken care of during post processing ?

Also, to make predictions on test data of dimensions 200 * 1024 * 1024, It took over 30min. Is this normal or could it be expedited ?

I am attaching the config file as well. Thanks a lot once again for the help.

3d_volEM_segmentation.yaml.txt

[danifranco]

Hi,

Let me answer you point by point:

This is clearly overfitting and can only be fixed with more training data, right ?

Yeah, seems to be overfitting. You can try to increase some of the data augmentation ranges, like brightness or contrast.

Looking at your script I would suggest increasing the DATA.PATCH_SIZE in XY axis, like 40x128x128 or even 20x256x256. Most of the mitochondria are within 20 slices, and the model will have more context, so I think it will perform better.

Also, to make predictions on test data of dimensions 20010241024, It took over 30min. Is this normal or could it be expedited ?

What takes a lot of time is a painting process to debug the results. You can disable it setting TEST.MATCHING_STATS_THS_COLORED_IMG to an empty list [ ] . This is the line in the config.py to that variable. This option creates a new image depicting the matching done between the prediction and the ground truth (it is stored in a folder called instance_associations). The instances painted in blue will be the FP, the TP are in green and the FN in red. The instance metrics consider a TP when the predicted instance matches the ground truth one with an IoU more than a certain threshold. The default is to measure three different thresholds: 0.3, 0.5 and 0.75 of IoU. It is determined by TEST.MATCHING_STATS_THS (link). The results are the lines that appear in the output as "DatasetMatching:". This option is super useful to debug the results, but it's true that it is slow, as the instances are painted one by one. I was thinking of disabling this option in the default settings. Since I know it can be a bit annoying for others like me, I'll go ahead and turn it off.

In case you want to know more about the metric you can find more info in MitoEM's second paper: Current Progress and Challenges in Large-Scale 3D Mitochondria Instance Segmentation

[danifranco]

Hi @Aseem139 , have you had time to check what we suggested? Can we close the issue?

[Aseem139]

Hi @danifranco Thanks a lot for all your help. I haven't yet had the time to train more models. We can close this and if something comes up I will open another issue. thanks a lot