Visualize activations functions using keras

December 22, 2017

(This article was originally published at SAS Programming for Data Mining, and syndicated at StatsBlogs.)

In keras, we can visualize activation functions' geometric properties using backend functions over layers of a model.

We all know the exact function of popular activation functions such as 'sigmoid', 'tanh', 'relu', etc, and we can feed data to these functions to directly obtain their output. But how to do that via keras without explicitly specifying their functional forms?

This can be done following the four steps below:

1. define a simple MLP model with a one dimension input data, a one neuron dense network as the hidden layer, and the output layer will have a 'linear' activation function for one neuron.
2. Extract layers' output of the model (fitted or not) via iterating through model.layers
3. Using backend function K.function() to obtain calculated output for a given input data
4. Feed desired data to the above functions to obtain the output from appropriate activation function.

The code below is a demo:

from keras.layers import Dense, Activation
from keras.models import Sequential
import keras.backend as K
import numpy as np
import matplotlib.pyplot as plt

def NNmodel(activationFunc='linear'):
Define a neural network which can be arbitrary but we keep it simple here
model = Sequential()
model.add(Dense(1, input_shape=(1,), activation=activationFunc))

model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adagrad', metrics=['mse'])
return model

def VisualActivation(activationFunc='relu', plot=True):
x = (np.arange(100)-50)/10
y = np.log(x+51/10)

model = NNmodel(activationFunc = activationFunc)
# optional to fit the model. If not fitted, initialized parameters will be used, x, epochs=1, batch_size=128, verbose=0)

# define the computing process
inX = model.input
outputs = [layer.output for layer in model.layers]
functions = [K.function([inX], [out]) for out in outputs]

# compute based on original inputs
for i in range(100):
test = x[i].reshape(-1, 1)
layer_outs = [func([test]) for func in functions]
activationLayer[i] = layer_outs[0][0][0][0]

# process results
activationDf = pd.DataFrame.from_dict(activationLayer, orient='index')
result=pd.concat([pd.DataFrame(x), activationDf], axis=1)
result.columns=['X', 'Activated']
result.set_index('X', inplace=True)
if plot:

return result

# Now we can visualize them (assuming default settings) :
actFuncs = ['sigmoid', 'tanh', 'hard_sigmoid', 'softplus', 'selu', 'elu']

figure = plt.figure()
for i, f in enumerate(actFuncs):
figure.add_subplot(3, 2, i+1)
out=VisualActivation(activationFunc=f, plot=False)
plt.plot(out.index, out.Activated)

This figure is the output from above code. As we can see, the geometric property of each activation function is well captured.

Please comment on the article here: SAS Programming for Data Mining

Tags: , ,