Callbacks and Utilities

A callback is a set of functions under astroNN.nn.callbacks and astroNN.nn.utilities modules to be applied at given stages of the training procedure. astroNN provides some customized callbacks which built on tensorflow.keras. You can just treat astroNN customized callbacks as conventional Keras callbacks.

astroNN also contains some handy utilities for data processing

Virtual CSVLogger (Callback)

class astroNN.nn.callbacks.VirutalCSVLogger(filename='training_history.csv', separator=',', append=False)[source]

A modification of keras’ CSVLogger, but not actually write a file until you call method to save

Parameters:
  • filename (str) – filename of the log to be saved on disk

  • separator (str) – separator of fields

  • append (bool) – whether allow append or not

Returns:

callback instance

Return type:

object

History:
2018-Feb-22 - Written - Henry Leung (University of Toronto)
2018-Mar-12 - Update - Henry Leung (University of Toronto)
savefile(folder_name=None)[source]

the method to actually save the file to disk

Parameters:

folder_name (Union[NoneType, str]) – foldername, can be None to save to current directory

VirutalCSVLogger is basically Keras’s CSVLogger without Python 2 support and won’t write the file to disk until savefile() method is called after the training where Keras’s CSVLogger will write to disk immediately.

VirutalCSVLogger can be imported by

1from astroNN.nn.callbacks import VirutalCSVLogger

It can be used with Keras, you just have to import the function from astroNN

 1def keras_model():
 2    # Your keras_model define here
 3    return model
 4
 5# Create a Virtual_CSVLogger instance first
 6csvlogger = VirutalCSVLogger()
 7
 8# Default filename is training_history.csv
 9# You have to set filename first before passing to Keras
10csvlogger.filename = 'training_history.csv'
11
12model = keras_model()
13model.compile(....)
14
15model.fit(...,callbacks=[csvlogger])
16
17# Save the file to current directory
18csvlogger.savefile()
19
20# OR to save the file to other directory
21csvlogger.savefile(folder_name='some_folder')

Raising Error on Nan (Callback)

class astroNN.nn.callbacks.ErrorOnNaN(monitor='loss')[source]

Callback that raise error when a NaN is encountered.

Returns:

callback instance

Return type:

object

History:
2018-May-07 - Written - Henry Leung (University of Toronto)
2021-Apr-22 - Written - Henry Leung (University of Toronto)

ErrorOnNaN is basically Keras’s TerminateOnNaN but will raise ValueError on Nan, its useful for python unittest to make sure you can catch the error and know something is wrong.

Normalizer (Utility)

astroNN Normalizer is called when train() method is called and involved pre_training_checklist_master() method defined in NeuralNetMaster Class. Normalizer will not normalize data/labels equal to magicnumber defined in configuration file. So that astroNN loss function can recognize those missing/bad data.

Normalizer consists of a few modes that you can, but the mode will minus mean and divide standard derivation to the data.

\[\text{Normalized Data} = \frac{\text{Data} - \text{Mean}}{\text{Standard Derivation}} \text{for Data} \neq \text{Magic Number}\]
  1. Mode 0 means normalizing data with mean=0 and standard derivation=1 (same as doing nothing)

1# If we have some data
2data = np.array([[1,2,3], [9,8,7]])
3
4# THe normalized data, mean std will as follow by this mode
5norm_data = array([[1,2,3], [9,8,7]])
6# the mean and standard derivation used to do the normalization
7mean = [0.]
8std = [1.]
  1. Mode 1 means normalizing data with a single mean and a single standard derivation of the data

1# If we have some data
2data = np.array([[1,2,3], [9,8,7]])
3
4# THe normalized data, mean std will as follow by this mode
5norm_data = array([[-1.28653504, -0.96490128, -0.64326752], [ 1.28653504,  0.96490128,  0.64326752]])
6# the mean and standard derivation used to do the normalization
7mean = [5.0]
8std = [3.11]
  1. Mode 2 means normalizing data with pixelwise means and pixelwise standard derivations of the data

1# If we have some data
2data = np.array([[1,2,3], [9,8,7]])
3
4# THe normalized data, mean std will as follow by this mode
5norm_data = array([[-4., -3., -2.], [ 4.,  3.,  2.]])
6# the mean and standard derivation used to do the normalization
7mean = [5., 5., 5.]
8std = [4., 3., 2.]
  1. Mode 3 means normalizing data with featurewise mean and standard derivation=1 the data (only centered the data), it is useful for normalizing spectra

1# If we have some data
2data = array([[1,2,3], [9,8,7]])
3
4# THe normalized data, mean std will as follow by this mode
5norm_data = array([[-1., -1., -1.], [ 1.,  1.,  1.]])
6# the mean and standard derivation used to do the normalization
7mean = [5., 5., 5.]
8std = [1.]
  1. Mode 3s means normalizing data with featurewise mean and standard derivation=1 the data (only centered the data), then apply sigmoid for normalization or sigmoid inverse for denormalization. It is useful for normalizing spectra for Variational Autoencoder with Negative Log Likelihood objective.

  2. Mode 255 means normalizing data with mean=127.5 and standard derivation=127.5, this mode is designed to normalize 8bit images

1# If we have some data
2data = np.array([[255,125,100], [99,87,250]])
3
4# THe normalized data, mean std will as follow by this mode
5norm_data = array([[ 1. , -0.01960784, -0.21568627], [-0.22352941, -0.31764706,  0.96078431]])
6# the mean and standard derivation used to do the normalization
7mean = [127.5]
8std = [127.5]

You can set the mode from a astroNN neural net instance before called train() method by

1# To set the normalization mode for input and labels
2astronn_neuralnet.input_norm_mode = ...
3astronn_neuralnet.labels_norm_mode = ...

You can use Normalizer() independently to take advantage of this function won’t touch data equal magicnumber. Normalizer() always return you the normalized data, the mean and standard derivation used to do the normalization

 1from astroNN.nn.utilities.normalizer import Normalizer
 2import numpy as np
 3
 4# Make some data up
 5data = np.array([[1.,2.,3.], [9.,8.,7.]])
 6
 7# Setup a normalizer instance with a mode, lets say mode 1
 8normer = Normalizer(mode=1)
 9
10# Use the instance method normalize to normalize the data
11norm_data = normer.normalize(data)
12
13print(norm_data)
14>>> array([[-1.28653504, -0.96490128, -0.64326752], [ 1.28653504,  0.96490128,  0.64326752]])
15print(normer.mean_labels)
16>>> 5.0
17print(normer.std_labels)
18>>> 3.1091263510296048
19
20# You can use the same instance (with same mean and std and mode) to demoralize data
21denorm_data = normer.denormalize(data)
22
23print(denorm_data)
24>>> array([[1.,2.,3.], [9.,8.,7.]])

Useful Handy Tensorflow function - astroNN.nn

astroNN.nn.reduce_var(x, axis=None, keepdims=False)[source]

Calculate variance using Tensorflow (as opposed to tf.nn.moment which return both variance and mean)

Parameters:
  • x (tf.Tensor) – Data

  • axis (int) – Axis

  • keepdims (boolean) – Keeping variance dimension as data or not

Returns:

Variance

Return type:

tf.Tensor

History:

2018-Mar-04 - Written - Henry Leung (University of Toronto)

astroNN.nn.intpow_avx2(x, n)[source]

Calculate integer power of float (including negative) even with Tensorflow compiled with AVX2 since –fast-math compiler flag aggressively optimize float operation which is common with AVX2 flag

Parameters:
  • x (tf.Tensor) – identifier

  • n (int) – an integer power (a float will be casted to integer!!)

Returns:

powered float(s)

Return type:

tf.Tensor

History:

2018-Aug-13 - Written - Henry Leung (University of Toronto)

 1from astroNN.nn import intpow_avx2
 2import tensorflow as tf
 3
 4print(intpow_avx2(tf.constant([-1.2]), 2))
 5>>> tf.Tensor([1.44], shape=(1,), dtype=float32)
 6
 7print(tf.pow(tf.constant([-1.2]), 2))
 8# if your tensorflow is compiled with AVX2 or --fast-math
 9>>> tf.Tensor([nan], shape=(1,), dtype=float32)
10# if your tensorflow is NOT compiled with AVX2 or --fast-math
11>>> tf.Tensor([1.44], shape=(1,), dtype=float32)

NumPy Implementation of Tensorflow function - astroNN.nn.numpy

astroNN has some handy numpy implementation of a number of tensorflow functions. The list of available functions are

astroNN.nn.numpy.kl_divergence(x, y)[source]

NumPy implementation of tf.distributions.kl_divergence

Either both x and y are ndarray or both x and y are astropy.Quatity, return without astropy units in all case

Parameters:
  • x (Union[ndarray, float]) – prediction

  • y (Union[ndarray, float]) – ground truth

Returns:

KL-divergence

Return type:

Union[ndarray, float]

History:

2018-May-13 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.l1(x, l1=0.0)[source]

NumPy implementation of tf.keras.regularizers.l1

Parameters:
  • x (Union[ndarray, float]) – Data to have L1 regularization coefficient calculated

  • l1 (Union[ndarray, float]) – L1 regularization parameter

Returns:

L1 regularization coefficient

Return type:

Union[ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.l2(x, l2=0.0)[source]

NumPy implementation of tf.keras.regularizers.l2

Parameters:
  • x (Union[ndarray, float]) – Data to have L2 regularization coefficient calculated

  • l2 (Union[ndarray, float]) – L2 regularization parameter

Returns:

L2 regularization coefficient

Return type:

Union[ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.mean_absolute_error(x, y, axis=None)[source]

NumPy implementation of tf.keras.metrics.mean_absolute_error with capability to deal with magicnumber and astropy Quantity

Either both x and y are ndarray or both x and y are astropy.Quatity, return without astropy units in all case

Parameters:
  • x (Union[ndarray, float, astropy.Quatity]) – prediction

  • y (Union[ndarray, float, astropy.Quatity]) – ground truth

  • axis (Union[NoneType, int]) – NumPy axis

Raise:

TypeError when only either x or y contains astropy units. Both x, y should carry/not carry astropy units at the same time

Returns:

Mean Absolute Error

Return type:

Union[ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.mean_absolute_percentage_error(x, y, axis=None)[source]
NumPy implementation of tf.keras.metrics.mean_absolute_percentage_error with capability to deal with magicnumber and astropy Quantity
Either both x and y are ndarray or both x and y are astropy.Quatity, return has no astropy units in all case
Parameters:
  • x (Union[ndarray, float, astropy.Quatity]) – prediction

  • y (Union[ndarray, float, astropy.Quatity]) – ground truth

  • axis (Union[NoneType, int]) – NumPy axis

Raise:

TypeError when only either x or y contains astropy units. Both x, y should carry/not carry astropy units at the same time

Returns:

Mean Absolute Percentage Error

Return type:

Union[ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.median_absolute_error(x, y, axis=None)[source]

NumPy implementation of a median version of tf.keras.metrics.mean_absolute_error with capability to deal with magicnumber and astropy Quantity

Either both x and y are ndarray or both x and y are astropy.Quatity, return without astropy units in all case

Parameters:
  • x (Union[ndarray, float, astropy.Quatity]) – prediction

  • y (Union[ndarray, float, astropy.Quatity]) – ground truth

  • axis (Union[NoneType, int]) – NumPy axis

Raise:

TypeError when only either x or y contains astropy units. Both x, y should carry/not carry astropy units at the same time

Returns:

Median Absolute Error

Return type:

Union[ndarray, float]

History:

2018-May-13 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.median_absolute_percentage_error(x, y, axis=None)[source]
NumPy implementation of a median version of tf.keras.metrics.mean_absolute_percentage_error with capability to
deal with magicnumber and astropy Quantity
Either both x and y are ndarray or both x and y are astropy.Quatity, return has no astropy units in all case
Parameters:
  • x (Union[ndarray, float, astropy.Quatity]) – prediction

  • y (Union[ndarray, float, astropy.Quatity]) – ground truth

  • axis (Union[NoneType, int]) – NumPy axis

Raise:

TypeError when only either x or y contains astropy units. Both x, y should carry/not carry astropy units at the same time

Returns:

Median Absolute Percentage Error

Return type:

Union[ndarray, float]

History:

2018-May-13 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.relu(x)[source]

NumPy implementation of tf.nn.relu

Parameters:

x (Union[ndarray, float]) – Data to have ReLU activated

Returns:

ReLU activated data

Return type:

Union[ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.sigmoid(x)[source]

NumPy implementation of tf.sigmoid, mask magicnumber

Parameters:

x (Union[ndarray, float]) – Data to be applied sigmoid activation

Returns:

Sigmoid activated data

Return type:

Union[ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)

astroNN.nn.numpy.sigmoid_inv(x)[source]

NumPy implementation of tf.sigmoid inverse, mask magicnumber

Parameters:

x (Union[numpy.ndarray, float]) – Data to be applied inverse sigmoid activation

Returns:

Inverse Sigmoid activated data

Return type:

Union[numpy.ndarray, float]

History:

2018-Apr-11 - Written - Henry Leung (University of Toronto)