Loss Functions and Metrics - astroNN.nn.losses¶

astroNN provides modified loss functions which are capable to deal with incomplete labels which are represented by magicnumber in astroNN configuration file or Magic Number in equations below. Since they are built on Tensorflow and follows Keras API requirement, all astroNN loss functions are fully compatible with Keras with Tensorflow backend, as well as directly be imported and used with Tensorflow, for most loss functions, the first argument is ground truth tensor and the second argument is prediction tensor from neural network.

Note

Always make sure when you are normalizing your data, keep the magic number as magic number. If you use astroNN normalizer, astroNN will take care of that.

Here are some explanations on variables in the following loss functions:

\(y_i\) means the ground truth labels, always represented by python variable y_true in astroNN

\(\hat{y_i}\) means the prediction from neural network, always represented by python variable y_pred in astroNN

Correction Term for Magic Number¶

astroNN.nn.magic_correction_term(y_true)¶

Calculate a correction term to prevent the loss being lowered by magic_num

Parameters:	y_true (tf.Tensor) – Ground Truth
Returns:	Correction Term
Return type:	tf.Tensor
History:	2018-Jan-30 - Written - Henry Leung (University of Toronto) 2018-Feb-17 - Updated - Henry Leung (University of Toronto)

Since astroNN deals with magic number by assuming the prediction from neural network for those ground truth with Magic Number is right, so we need a correction term.

The correction term in astroNN is defined by the following equation and we call the equation \(\mathcal{F}_{correction}\)

\[\mathcal{F}_{correction} = \frac{\text{Non-Magic Number Count} + \text{Magic Number Count}}{\text{Non Magic Number Count}}\]

In case of no labels with Magic Number is presented, \(\mathcal{F}_{correction}\) will equal to 1

Mean Squared Error¶

astroNN.nn.losses.mean_squared_error(y_true, y_pred)¶

Calculate mean square error losses

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Mean Squared Error
Return type:	tf.Tensor
History:	2017-Nov-16 - Written - Henry Leung (University of Toronto)

MSE is based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} (\hat{y_i}-y_i)^2 & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=mean_squared_error, ...)

Mean Absolute Error¶

astroNN.nn.losses.mean_absolute_error(y_true, y_pred)¶

Calculate mean absolute error, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Mean Absolute Error
Return type:	tf.Tensor
History:	2018-Jan-14 - Written - Henry Leung (University of Toronto)

MAE is based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} \left| \hat{y_i}-y_i \right| & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=mean_absolute_error, ...)

Mean Error¶

astroNN.nn.losses.mean_error(y_true, y_pred)¶

Calculate mean error as a way to get the bias in prediction, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Mean Error
Return type:	tf.Tensor
History:	2018-May-22 - Written - Henry Leung (University of Toronto)

Mean Error is a metrics to evaluate the bias of prediction and is based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} \hat{y_i}-y_i & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=mean_error, ...)

Regression Loss and Predictive Variance Loss for Bayesian Neural Net¶

astroNN.nn.losses.robust_mse(y_true, y_pred, variance, labels_err)¶

Calculate predictive variance, and takes account of labels error in Bayesian Neural Network

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction variance (Union(tf.Tensor, tf.Variable)) – Predictive Variance labels_err (Union(tf.Tensor, tf.Variable)) – Known labels error, give zeros if unknown/unavailable
Returns:	Robust Mean Squared Error, can be used directly with Tensorflow
Return type:	tf.Tensor
History:	2018-April-07 - Written - Henry Leung (University of Toronto)

astroNN.nn.losses.mse_lin_wrapper(var, labels_err)¶

Calculate predictive variance, and takes account of labels error in Bayesian Neural Network

Returned Funtion Parameter:
Parameters:	var (Union(tf.Tensor, tf.Variable)) – Predictive Variance labels_err (Union(tf.Tensor, tf.Variable)) – Known labels error, give zeros if unknown/unavailable
Returns:	Robust MSE function for labels prediction neurones, which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Prediction Return (tf.Tensor): Robust Mean Squared Error
History:	2017-Nov-16 - Written - Henry Leung (University of Toronto)

astroNN.nn.losses.mse_var_wrapper(lin, labels_err)¶

Calculate predictive variance, and takes account of labels error in Bayesian Neural Network

Returned Funtion Parameter:
Parameters:	lin (Union(tf.Tensor, tf.Variable)) – Prediction labels_err (Union(tf.Tensor, tf.Variable)) – Known labels error, give zeros if unknown/unavailable
Returns:	Robust MSE function for predictive variance neurones which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Predictive Variance Return (tf.Tensor): Robust Mean Squared Error
History:	2017-Nov-16 - Written - Henry Leung (University of Toronto)

It is based on the equation implemented as robust_mse(), please notice \(s_i\) is representing \(log((\sigma_{predictive, i})^2 + (\sigma_{known, i})^2)\). Neural network not predicting variance directly to avoid numerical instability but predicting \(log((\sigma_{i})^2)\)

\[\begin{split}Loss_i = \begin{cases} \begin{split} \frac{1}{2} (\hat{y_i}-y_i)^2 e^{-s_i} + \frac{1}{2}(s_i) & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{BNN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

They basically do the same things and can be used with Keras, you just have to import the functions from astroNN

def keras_model():
    # Your keras_model define here

    # model for the training process
    model = Model(inputs=[input_tensor, labels_err_tensor], outputs=[output, variance_output])

    # model for the prediction
    model_prediction = Model(inputs=input_tensor, outputs=[output, variance_output])

    variance_output = Dense(name='variance_output', ...)
    output = Dense(name='output', ...)

    predictive_variance_loss = mse_var_wrapper(output, labels_err_tensor)
    output_loss = mse_lin_wrapper(predictive_variance, labels_err_tensor)

    return model, model_prediction, output_loss, predictive_variance_loss

model, model_prediction, output_loss, predictive_variance_loss = keras_model()
# remember to import astroNN loss function first
model.compile(loss={'output': output_loss, 'variance_output': predictive_variance_loss}, ...)

To better understand this loss function, you can see the following plot of Loss vs Variance colored by squared difference which is \((\hat{y_i}-y_i)^2\)

Mean Squared Logarithmic Error¶

astroNN.nn.losses.mean_squared_logarithmic_error(y_true, y_pred)¶

Calculate mean squared logarithmic error, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Mean Squared Logarithmic Error
Return type:	tf.Tensor
History:	2018-Feb-17 - Written - Henry Leung (University of Toronto)

MSLE will first clip the values of prediction from neural net for the sake of numerical stability,

\[ \begin{align}\begin{aligned}\begin{split}y_i = \begin{cases} \begin{split} \epsilon + 1 & \text{ for } y_i < \epsilon \\ y_i + 1 & \text{ for otherwise } \end{split} \end{cases}\end{split}\\\text{where } \epsilon \text{ is a small constant}\end{aligned}\end{align} \]

Then MSLE is based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} (\log{(\hat{y_i})} - \log{(y_i)})^2 & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=mean_squared_logarithmic_error, ...)

Mean Absolute Percentage Error¶

astroNN.nn.losses.mean_absolute_percentage_error(y_true, y_pred)¶

Calculate mean absolute percentage error, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Mean Absolute Percentage Error
Return type:	tf.Tensor
History:	2018-Feb-17 - Written - Henry Leung (University of Toronto)

Mean Absolute Percentage Error will first clip the values of prediction from neural net for the sake of numerical stability,

\[ \begin{align}\begin{aligned}\begin{split}y_i = \begin{cases} \begin{split} \epsilon & \text{ for } y_i < \epsilon \\ y_i & \text{ for otherwise } \end{split} \end{cases}\end{split}\\\text{where } \epsilon \text{ is a small constant}\end{aligned}\end{align} \]

Then Mean Absolute Percentage Error is based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} 100 \text{ } \frac{\left| y_i - \hat{y_i} \right|}{y_i} & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=mean_absolute_percentage_error, ...)

Mean Percentage Error¶

astroNN.nn.losses.mean_percentage_error(y_true, y_pred)¶

Calculate mean percentage error, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Mean Percentage Error
Return type:	tf.Tensor
History:	2018-Jun-06 - Written - Henry Leung (University of Toronto)

Mean Percentage Error will first clip the values of prediction from neural net for the sake of numerical stability,

\[ \begin{align}\begin{aligned}\begin{split}y_i = \begin{cases} \begin{split} \epsilon & \text{ for } y_i < \epsilon \\ y_i & \text{ for otherwise } \end{split} \end{cases}\end{split}\\\text{where } \epsilon \text{ is a small constant}\end{aligned}\end{align} \]

Then Mean Percentage Error is based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} 100 \text{ } \frac{y_i - \hat{y_i}}{y_i} & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=mean_percentage_error, ...)

Categorical Cross-Entropy¶

astroNN.nn.losses.categorical_crossentropy(y_true, y_pred, from_logits=False)¶

Categorical cross-entropy between an output tensor and a target tensor, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction from_logits (boolean) – From logits space or not. If you want to use logits, please use from_logits=True
Returns:	Categorical Cross-Entropy
Return type:	tf.Tensor
History:	2018-Jan-14 - Written - Henry Leung (University of Toronto)

Categorical Cross-Entropy will first clip the values of prediction from neural net for the sake of numerical stability if the prediction is not coming from logits (before softmax activated)

\[ \begin{align}\begin{aligned}\begin{split}\hat{y_i} = \begin{cases} \begin{split} \epsilon & \text{ for } \hat{y_i} < \epsilon \\ 1 - \epsilon & \text{ for } \hat{y_i} > 1 - \epsilon \\ \hat{y_i} & \text{ for otherwise } \end{split} \end{cases}\end{split}\\\text{where } \epsilon \text{ is a small constant}\end{aligned}\end{align} \]

and then based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} y_i \log{(\hat{y_i})} & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = - \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=categorical_crossentropy(from_logits=False), ...)

Binary Cross-Entropy¶

astroNN.nn.losses.binary_crossentropy(y_true, y_pred, from_logits=False)¶

Binary cross-entropy between an output tensor and a target tensor, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction from_logits (boolean) – From logits space or not. If you want to use logits, please use from_logits=True
Returns:	Binary Cross-Entropy
Return type:	tf.Tensor
History:	2018-Jan-14 - Written - Henry Leung (University of Toronto)

Binary Cross-Entropy will first clip the values of prediction from neural net for the sake of numerical stability if from_logits=False

\[ \begin{align}\begin{aligned}\begin{split}\hat{y_i} = \begin{cases} \begin{split} \epsilon & \text{ for } \hat{y_i} < \epsilon \\ 1 - \epsilon & \text{ for } \hat{y_i} > 1 - \epsilon \\ \hat{y_i} & \text{ for otherwise } \end{split} \end{cases}\end{split}\\\text{where } \epsilon \text{ is a small constant}\end{aligned}\end{align} \]

and then based on the equation

\[\begin{split}Loss_i = \begin{cases} \begin{split} y_i \log{(\hat{y_i})} + (1-y_i)\log{(1-\hat{y_i})} & \text{ for } y_i \neq \text{Magic Number}\\ \hat{y_i} \log{(\hat{y_i})} + (1-\hat{y_i})\log{(1-\hat{y_i})} & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

to avoid numerical instability if from_logits=True, we can reformulate it as

\[\begin{split}Loss_i = \begin{cases} \begin{split} \max{(\hat{y_i}, 0)} - y_i \hat{y_i} + \log{(1+e^{-\|\hat{y_i}\|})} & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

And thus the loss for mini-batch is

\[Loss_{NN} = - \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=binary_crossentropy(from_logits=False), ...)

Categorical Cross-Entropy and Predictive Logits Variance for Bayesian Neural Net¶

astroNN.nn.losses.robust_categorical_crossentropy(y_true, y_pred, logit_var)¶

Calculate categorical accuracy, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction in logits space logit_var (Union(tf.Tensor, tf.Variable)) – Predictive variance in logits space
Returns:	categorical cross-entropy
Return type:	tf.Tensor
History:	2018-Mar-15 - Written - Henry Leung (University of Toronto)

astroNN.nn.losses.bayesian_categorical_crossentropy_wrapper(logit_var)¶

Categorical crossentropy between an output tensor and a target tensor for Bayesian Neural Network

equation (12) of arxiv:1703.04977

Returned Function Parameter:
Parameters:	logit_var (Union(tf.Tensor, tf.Variable)) – Predictive variance
Returns:	Robust categorical_crossentropy function for predictive variance neurones which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Prediction in logits space Return (tf.Tensor): Robust categorical crossentropy
History:	2018-Mar-15 - Written - Henry Leung (University of Toronto)

astroNN.nn.losses.bayesian_categorical_crossentropy_var_wrapper(logits)¶

Categorical crossentropy between an output tensor and a target tensor for Bayesian Neural Network

equation (12) of arxiv:1703.04977

Returned Function Parameter:
Parameters:	logits (Union(tf.Tensor, tf.Variable)) – Prediction in logits space
Returns:	Robust categorical_crossentropy function for predictive variance neurones which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Predictive variance in logits space Return (tf.Tensor): Robust categorical crossentropy
History:	2018-Mar-15 - Written - Henry Leung (University of Toronto)

It is based on Equation 12 from arxiv:1703.04977. \(s_i\) is representing the predictive variance of logits

\[\begin{split}Loss_i = \begin{cases} \begin{split} \text{Categorical Cross-Entropy} + \text{Distorted Categorical Cross-Entropy} + e^{s_i} - 1 & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

where Distorted Categorical Cross-Entropy is defined as

\[\text{elu} (\text{Categorical Cross-Entropy}(y_i, \hat{y_i}) - \text{Categorical Cross-Entropy}(y_i, \mathcal{N}(\hat{y_i}, \sqrt{s_i})))\]

And thus the loss for mini-batch is

\[Loss_{BNN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

bayesian_categorical_crossentropy_wrapper is for the prediction neurones

bayesian_categorical_crossentropy_var_wrapper is for the predictive variance neurones

They basically do the same things and can be used with Keras, you just have to import the functions from astroNN

def keras_model():
    # Your keras_model define here

    # model for the training process
    model = Model(inputs=[input_tensor], outputs=[output, variance_output])

    # model for the prediction
    model_prediction = Model(inputs=input_tensor, outputs=[output, variance_output])

    variance_output = Dense(name='predictive_variance', ...)
    output = Dense(name='output', ...)

    predictive_variance_loss = bayesian_categorical_crossentropy_var_wrapper(output)
    output_loss = bayesian_categorical_crossentropy_wrapper(predictive_variance)

    return model, model_prediction, output_loss, predictive_variance_loss

model, model_prediction, output_loss, predictive_variance_loss = keras_model()
# remember to import astroNN loss function first
model.compile(loss={'output': output_loss, 'variance_output': predictive_variance_loss}, ...)

Binary Cross-Entropy and Predictive Logits Variance for Bayesian Neural Net¶

astroNN.nn.losses.robust_binary_crossentropy(y_true, y_pred, logit_var)¶

Calculate binary accuracy, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction in logits space logit_var (Union(tf.Tensor, tf.Variable)) – Predictive variance in logits space
Returns:	categorical cross-entropy
Return type:	tf.Tensor
History:	2018-Mar-15 - Written - Henry Leung (University of Toronto)

astroNN.nn.losses.bayesian_binary_crossentropy_wrapper(logit_var)¶

Binary crossentropy between an output tensor and a target tensor for Bayesian Neural Network

equation (12) of arxiv:1703.04977

Returned Function Parameter:
Parameters:	logit_var (Union(tf.Tensor, tf.Variable)) – Predictive variance
Returns:	Robust binary_crossentropy function for predictive variance neurones which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Prediction in logits space Return (tf.Tensor): Robust binary crossentropy
History:	2018-Mar-15 - Written - Henry Leung (University of Toronto)

astroNN.nn.losses.bayesian_binary_crossentropy_var_wrapper(logits)¶

Binary crossentropy between an output tensor and a target tensor for Bayesian Neural Network

equation (12) of arxiv:1703.04977

Returned Function Parameter:
Parameters:	logits (Union(tf.Tensor, tf.Variable)) – Prediction in logits space
Returns:	Robust binary_crossentropy function for predictive variance neurones which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Predictive variance in logits space Return (tf.Tensor): Robust binary crossentropy
History:	2018-Mar-15 - Written - Henry Leung (University of Toronto)

It is based on Equation 12 from arxiv:1703.04977. \(s_i\) is representing the predictive variance of logits

\[\begin{split}Loss_i = \begin{cases} \begin{split} \text{Binary Cross-Entropy} + \text{Distorted Binary Cross-Entropy} + e^{s_i} - 1 & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

where Distorted Binary Cross-Entropy is defined as

\[\text{elu} (\text{Binary Cross-Entropy}(y_i, \hat{y_i}) - \text{Binary Cross-Entropy}(y_i, \mathcal{N}(\hat{y_i}, \sqrt{s_i})))\]

And thus the loss for mini-batch is

\[Loss_{BNN} = \frac{1}{D} \sum_{i=1}^{batch} (Loss_i \mathcal{F}_{correction, i})\]

bayesian_binary_crossentropy_wrapper is for the prediction neurones

bayesian_binary_crossentropy_var_wrapper is for the predictive variance neurones

They basically do the same things and can be used with Keras, you just have to import the functions from astroNN

def keras_model():
    # Your keras_model define here

    # model for the training process
    model = Model(inputs=[input_tensor], outputs=[output, variance_output])

    # model for the prediction
    model_prediction = Model(inputs=input_tensor, outputs=[output, variance_output])

    variance_output = Dense(name='predictive_variance', ...)
    output = Dense(name='output', ...)

    predictive_variance_loss = bayesian_binary_crossentropy_var_wrapper(output)
    output_loss = bayesian_binary_crossentropy_wrapper(predictive_variance)

    return model, model_prediction, output_loss, predictive_variance_loss

model, model_prediction, output_loss, predictive_variance_loss = keras_model()
# remember to import astroNN loss function first
model.compile(loss={'output': output_loss, 'variance_output': predictive_variance_loss}, ...)

Categorical Classification Accuracy¶

astroNN.nn.losses.categorical_accuracy(y_true, y_pred)¶

Calculate categorical accuracy, ignoring the magic number

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Categorical Classification Accuracy
Return type:	tf.Tensor
History:	2018-Jan-21 - Written - Henry Leung (University of Toronto)

Categorical Classification Accuracy will first deal with Magic Number

\[\begin{split}Loss_i = \begin{cases} \begin{split} y_i & \text{ for } y_i \neq \text{Magic Number}\\ 0 & \text{ for } y_i = \text{Magic Number} \end{split} \end{cases}\end{split}\]

Then based on the equation

\[\begin{split}Accuracy_i = \begin{cases} \begin{split} 1 & \text{ for } \text{Argmax}(y_i) = \text{Argmax}(\hat{y_i})\\ 0 & \text{ for } \text{Argmax}(y_i) \neq \text{Argmax}(\hat{y_i}) \end{split} \end{cases}\end{split}\]

And thus the accuracy for is

\[Accuracy = \frac{1}{D} \sum_{i=1}^{labels} (Accuracy_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's metrics function first
model.compile(metrics=categorical_accuracy, ...)

Note

Please make sure you use categorical_accuracy when using categorical_crossentropy as the loss function

Binary Classification Accuracy¶

astroNN.nn.losses.binary_accuracy(from_logits=False)¶

Calculate binary accuracy, ignoring the magic number

Returned Funtion Parameter:
Parameters:	from_logits (boolean) – From logits space or not. If you want to use logits, please use from_logits=True
Returns:	Function for Binary classification accuracy which matches Keras losses API
Return type:	function
	function(y_true, y_pred) - y_true (tf.Tensor): Ground Truth - y_pred (tf.Tensor): Prediction Return (tf.Tensor): Binary Classification Accuracy
History:	2018-Jan-31 - Written - Henry Leung (University of Toronto)

Binary Classification Accuracy will round the values of prediction if from_logits=False or will apply sigmoid first and then round the values of prediction if from_logits=True

\[\begin{split}\hat{y_i} = \begin{cases} \begin{split} 1 & \text{ for } \hat{y_i} > 0.5 \\ 0 & \text{ for } \hat{y_i} \leq 0.5 \end{split} \end{cases}\end{split}\]

and then based on the equation

\[\begin{split}Accuracy_i = \begin{cases} \begin{split} 1 & \text{ for } y_i = \hat{y_i}\\ 0 & \text{ for } y_i \neq \hat{y_i} \end{split} \end{cases}\end{split}\]

And thus the accuracy for is

\[Accuracy = \frac{1}{D} \sum_{i=1}^{labels} (Accuracy_i \mathcal{F}_{correction, i})\]

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's metrics function first
model.compile(metrics=binary_accuracy(from_logits=False), ...)

Note

Please make sure you use binary_accuracy when using binary_crossentropy as the loss function

Zeros Loss¶

astroNN.nn.losses.zeros_loss(y_true, y_pred)¶

Always return zeros

Parameters:	y_true (Union(tf.Tensor, tf.Variable)) – Ground Truth y_pred (Union(tf.Tensor, tf.Variable)) – Prediction
Returns:	Zeros
Return type:	tf.Tensor
History:	2018-May-24 - Written - Henry Leung (University of Toronto)

zeros_loss is a loss function that will always return zero loss and the function matches Keras API. It is mainly designed to do testing or experiments.

It can be used with Keras, you just have to import the function from astroNN

def keras_model():
    # Your keras_model define here
    return model

model = keras_model()
# remember to import astroNN's loss function first
model.compile(loss=zeros_loss, ...)

Previous topic

Next topic

Loss Functions and Metrics - astroNN.nn.losses¶

Correction Term for Magic Number¶

Mean Squared Error¶

Mean Absolute Error¶

Mean Error¶

Regression Loss and Predictive Variance Loss for Bayesian Neural Net¶

Mean Squared Logarithmic Error¶

Mean Absolute Percentage Error¶

Mean Percentage Error¶

Categorical Cross-Entropy¶

Binary Cross-Entropy¶

Categorical Cross-Entropy and Predictive Logits Variance for Bayesian Neural Net¶

Binary Cross-Entropy and Predictive Logits Variance for Bayesian Neural Net¶

Categorical Classification Accuracy¶

Binary Classification Accuracy¶

Zeros Loss¶