neural network - caffe with multi-label images

Question

I have a dataset of images that have multiple labels; There are 100 classes in the dataset, and each image has 1 to 5 labels associated with them.

I'm following the instruction in the following URL:

https://github.com/BVLC/caffe/issues/550

It says that I need to generate a text file listing the images and its labels as in

/home/my_test_dir/picture-foo.jpg 0
/home/my_test_dir/picture-foo1.jpg 1

In my case, since I have multi-label images, does it work to simply add labels as in following?

/home/my_test_dir/picture-foo.jpg 0 2 5
/home/my_test_dir/picture-foo1.jpg 1 4

I have a feeling that it's probably not going to be that simple, and if I'm right, in what step and how should I integrate the multi-label-ness of the dataset in the process of setting up Caffe?

Answer 1

I believe Shai's answer is no longer up-to-date. Caffe supports multi-label/matrix ground truth for HDF5 and LMDB formats. The python snippet in this github comment demonstrates how to construct multi-label LMDB ground truth (see Shai's answer for HDF5 format). Different from the construction of single-label image datasets, an lmdb is constructed for the images while a second separate lmdb is constructed for the multi-label ground truth data. The snippet deals with spatial multi-label ground truth useful for pixel-wise labeling of images.

The order in which data is written to the lmdb is crucial. The order of the ground truth must match the order of the images.

Loss layers such as SOFTMAX_LOSS, EUCLIDEAN_LOSS, SIGMOID_CROSS_ENTROPY_LOSS also support multi-label data. However, the Accuracy layer is still limited to single-label data. You might want to follow this github issue to keep track of when this feature is added to Caffe.