tensorflow - What is y_true and y_pred when creating a custom metric in Keras?

Question

I want to implement my custom metric in Keras. According to the documentation, my custom metric should be defined as a function that takes as input two tensors, y_pred and y_true, and returns a single tensor value.

However, I'm confused to what exactly will be contained in these tensors y_pred and y_true when the optimization is running. Is it just one data point? Is it the whole batch? The whole epoch (probably not)? Is there a way to obtain these tensors' shapes?

Can someone point to a trustworthy place where I can get this information? Any help would be appreciated. Not sure if relevant, but I'm using TensorFlow backend.

---

Things I tried so far, in order to answer this:

• Checking the Keras metrics documentation (no explanation there about what these tensors are).
• Checking the source code for the Keras metrics and trying to understand these tensors by looking at the Keras implementation for other metrics (This seems to suggest that y_true and y_pred have the labels for an entire batch, but I'm not sure).
• Reading these stackoverflow questions: 1, 2, 3, and others (none answer my question, most are centered on the OP not clearly understanding the difference between a tensor and the values computed using that tensor during the session).
• Printing the values of y_true and y_pred during the optimization, by defining a metric like this:

    def test_metric(y_true, y_pred):
        y_true = K.print_tensor(y_true)
        y_pred = K.print_tensor(y_pred)
        return y_true - y_pred

(unfortunately these don't print anything during the optimization).

Answer 1

y_true and y_pred

The tensor y_true is the true data (or target, ground truth) you pass to the fit method.
It's a conversion of the numpy array y_train into a tensor.

The tensor y_pred is the data predicted (calculated, output) by your model.

Usually, both y_true and y_pred have exactly the same shape. A few of the losses, such as the sparse ones, may accept them with different shapes.

---

The shape of y_true

It contains an entire batch. Its first dimension is always the batch size, and it must exist, even if the batch has only one element.

Two very easy ways to find the shape of y_true are:

• check your true/target data: print(Y_train.shape)
• check your model.summary() and see the last output

But its first dimension will be the batch size.

So, if your last layer outputs (None, 1), the shape of y_true is (batch, 1). If the last layer outputs (None, 200,200, 3), then y_true will be (batch, 200,200,3).

---

Custom metrics and loss functions

Unfotunately, printing custom metrics will not reveal their content (unless you are using eager mode on, and you have calculated every step of the model with data).
You can see their shapes with print(K.int_shape(y_pred)), for instance.

Remember that these libraries first "compile a graph", then later "runs it with data". When you define your loss, you're in the compile phase, and asking for data needs the model to run.

But even if the result of your metric is multidimensional, keras will automatically find ways to output a single scalar for that metric. (Not sure what is the operation, but very probably a K.mean() hidden under the table - it's interesting to return the entire batch, so Keras applies other operations such as sample weights, for instance).

Sources. After you get used to keras, this understanding gets natural from simply reading this part:

y_true: True labels. Theano/TensorFlow tensor.
y_pred: Predictions. Theano/TensorFlow tensor of the same shape as y_true.

True labels mean true/target data. Labels is a badly chosen word here, it is only really "labels" in classification models.
Predictions mean the results of your model.