It’s Sunday morning, it’s quiet and you wake up with a big smile on your face. Today is going to be a great day! Except, your phone rings, rather “internationally”. You pick it up slowly and hear something really bizarre - “Bonjour, je suis Michele. Oops, sorry. I am Michele, your personal bank agent.”.

What could possibly be so urgent for someone from Switzerland to call you at this hour? “Did you authorize a transaction for $3,358.65 for 100 copies of Diablo 3?” Immediately, you start thinking of ways to explain why you did that to your loved one. “No, I didn’t !?“. Michele’s answer is quick and to the point - “Thank you, we’re on it”.

Whew, that was close! But how did Michele knew that this transaction was suspicious? After all, you did order 10 new smartphones from that same bank account, last week - Michele didn’t call then.

Annual global fraud losses reached $21.8 billion in 2015, according to Nilson Report. Probably you feel very lucky if you are a fraud. About every 12 cents per $100 were stolen in the US during the same year. Our friend Michele might have a serious problem to solve here.

In this part of the series, we will train an Autoencoder Neural Network (implemented in Keras) in unsupervised (or semi-supervised) fashion for Anomaly Detection in credit card transaction data. The trained model will be evaluated on pre-labeled and anonymized dataset.

The source code and pre-trained model are available on GitHub here.

Setup

We will be using TensorFlow 1.2 and Keras 2.0.4. Let’s begin:

1import pandas as pd
2import numpy as np
3import pickle
4import matplotlib.pyplot as plt
5from scipy import stats
6import tensorflow as tf
7import seaborn as sns
8from pylab import rcParams
9from sklearn.model_selection import train_test_split
10from keras.models import Model, load_model
11from keras.layers import Input, Dense
12from keras.callbacks import ModelCheckpoint, TensorBoard
13from keras import regularizers
14
15%matplotlib inline
16
17sns.set(style='whitegrid', palette='muted', font_scale=1.5)
18
19rcParams['figure.figsize'] = 14, 8
20
21RANDOM_SEED = 42
22LABELS = ["Normal", "Fraud"]

Loading the data

The dataset we’re going to use can be downloaded from Kaggle. It contains data about credit card transactions that occurred during a period of two days, with 492 frauds out of 284,807 transactions.

All variables in the dataset are numerical. The data has been transformed using PCA transformation(s) due to privacy reasons. The two features that haven’t been changed are Time and Amount. Time contains the seconds elapsed between each transaction and the first transaction in the dataset.

1df = pd.read_csv("data/creditcard.csv")

Exploration

1df.shape

1(284807, 31)

31 columns, 2 of which are Time and Amount. The rest are output from the PCA transformation. Let’s check for missing values:

1df.isnull().values.any()

1False

1count_classes = pd.value_counts(df['Class'], sort = True)
2count_classes.plot(kind = 'bar', rot=0)
3plt.title("Transaction class distribution")
4plt.xticks(range(2), LABELS)
5plt.xlabel("Class")
6plt.ylabel("Frequency");

We have a highly imbalanced dataset on our hands. Normal transactions overwhelm the fraudulent ones by a large margin. Let’s look at the two types of transactions:

1frauds = df[df.Class == 1]
2normal = df[df.Class == 0]

1frauds.shape

1(492, 31)

1normal.shape

1(284315, 31)

How different are the amount of money used in different transaction classes?

1frauds.Amount.describe()

1count     492.000000
2mean      122.211321
3std       256.683288
4min         0.000000
525%         1.000000
650%         9.250000
775%       105.890000
8max      2125.870000
9Name: Amount, dtype: float64

1normal.Amount.describe()

1count    284315.000000
2mean         88.291022
3std         250.105092
4min           0.000000
525%           5.650000
650%          22.000000
775%          77.050000
8max       25691.160000
9Name: Amount, dtype: float64

Let’s have a more graphical representation:

1f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
2f.suptitle('Amount per transaction by class')
3
4bins = 50
5
6ax1.hist(frauds.Amount, bins = bins)
7ax1.set_title('Fraud')
8
9ax2.hist(normal.Amount, bins = bins)
10ax2.set_title('Normal')
11
12plt.xlabel('Amount ($)')
13plt.ylabel('Number of Transactions')
14plt.xlim((0, 20000))
15plt.yscale('log')
16plt.show();

Do fraudulent transactions occur more often during certain time?

1f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
2f.suptitle('Time of transaction vs Amount by class')
3
4ax1.scatter(frauds.Time, frauds.Amount)
5ax1.set_title('Fraud')
6
7ax2.scatter(normal.Time, normal.Amount)
8ax2.set_title('Normal')
9
10plt.xlabel('Time (in Seconds)')
11plt.ylabel('Amount')
12plt.show()

Doesn’t seem like the time of transaction really matters.

Autoencoders

Autoencoders can seem quite bizarre at first. The job of those models is to predict the input, given that same input. Puzzling? Definitely was for me, the first time I heard it.

More specifically, let’s take a look at Autoencoder Neural Networks. This autoencoder tries to learn to approximate the following identity function:

$\textstyle f_{W,b}(x) \approx x$

While trying to do just that might sound trivial at first, it is important to note that we want to learn a compressed representation of the data, thus find structure. This can be done by limiting the number of hidden units in the model. Those kind of autoencoders are called undercomplete.

Here’s a visual representation of what an Autoencoder might learn:

Reconstruction error

We optimize the parameters of our Autoencoder model in such way that a special kind of error - reconstruction error is minimized. In practice, the traditional squared error is often used:

$\textstyle L(x,x') = ||\, x - x'||^2$

If you want to learn more about Autoencoders I highly recommend the following videos by Hugo Larochelle:

Preparing the data

First, let’s drop the Time column (not going to use it) and use the scikit’s StandardScaler on the Amount. The scaler removes the mean and scales the values to unit variance:

1from sklearn.preprocessing import StandardScaler
2
3data = df.drop(['Time'], axis=1)
4
5data['Amount'] = StandardScaler().fit_transform(data['Amount'].values.reshape(-1, 1))

Training our Autoencoder is gonna be a bit different from what we are used to. Let’s say you have a dataset containing a lot of non fraudulent transactions at hand. You want to detect any anomaly on new transactions. We will create this situation by training our model on the normal transactions, only. Reserving the correct class on the test set will give us a way to evaluate the performance of our model. We will reserve 20% of our data for testing:

1X_train, X_test = train_test_split(data, test_size=0.2, random_state=RANDOM_SEED)
2X_train = X_train[X_train.Class == 0]
3X_train = X_train.drop(['Class'], axis=1)
4
5y_test = X_test['Class']
6X_test = X_test.drop(['Class'], axis=1)
7
8X_train = X_train.values
9X_test = X_test.values

1X_train.shape

1(227451, 29)

Building the model

Our Autoencoder uses 4 fully connected layers with 14, 7, 7 and 29 neurons respectively. The first two layers are used for our encoder, the last two go for the decoder. Additionally, L1 regularization will be used during training:

1input_dim = X_train.shape[1]
2encoding_dim = 14

1input_layer = Input(shape=(input_dim, ))
2
3encoder = Dense(encoding_dim, activation="tanh",
4                activity_regularizer=regularizers.l1(10e-5))(input_layer)
5encoder = Dense(int(encoding_dim / 2), activation="relu")(encoder)
6
7decoder = Dense(int(encoding_dim / 2), activation='tanh')(encoder)
8decoder = Dense(input_dim, activation='relu')(decoder)
9
10autoencoder = Model(inputs=input_layer, outputs=decoder)

Let’s train our model for 100 epochs with a batch size of 32 samples and save the best performing model to a file. The ModelCheckpoint provided by Keras is really handy for such tasks. Additionally, the training progress will be exported in a format that TensorBoard understands.

1nb_epoch = 100
2batch_size = 32
3
4autoencoder.compile(optimizer='adam',
5                    loss='mean_squared_error',
6                    metrics=['accuracy'])
7
8checkpointer = ModelCheckpoint(filepath="model.h5",
9                               verbose=0,
10                               save_best_only=True)
11tensorboard = TensorBoard(log_dir='/media/old-tf-hackers-7/logs',
12                          histogram_freq=0,
13                          write_graph=True,
14                          write_images=True)
15
16history = autoencoder.fit(X_train, X_train,
17                    epochs=nb_epoch,
18                    batch_size=batch_size,
19                    shuffle=True,
20                    validation_data=(X_test, X_test),
21                    verbose=1,
22                    callbacks=[checkpointer, tensorboard]).history

1autoencoder = load_model('model.h5')

Evaluation

1plt.plot(history['loss'])
2plt.plot(history['val_loss'])
3plt.title('model loss')
4plt.ylabel('loss')
5plt.xlabel('epoch')
6plt.legend(['train', 'test'], loc='upper right');

The reconstruction error on our training and test data seems to converge nicely. Is it low enough? Let’s have a closer look at the error distribution:

1predictions = autoencoder.predict(X_test)

1mse = np.mean(np.power(X_test - predictions, 2), axis=1)
2error_df = pd.DataFrame({'reconstruction_error': mse,
3                        'true_class': y_test})

1error_df.describe()

Reconstruction error without fraud

1fig = plt.figure()
2ax = fig.add_subplot(111)
3normal_error_df = error_df[(error_df['true_class']== 0) & (error_df['reconstruction_error'] < 10)]
4_ = ax.hist(normal_error_df.reconstruction_error.values, bins=10)

Reconstruction error with fraud

1fig = plt.figure()
2ax = fig.add_subplot(111)
3fraud_error_df = error_df[error_df['true_class'] == 1]
4_ = ax.hist(fraud_error_df.reconstruction_error.values, bins=10)

1from sklearn.metrics import (confusion_matrix, precision_recall_curve, auc,
2                             roc_curve, recall_score, classification_report, f1_score,
3                             precision_recall_fscore_support)

ROC curves are very useful tool for understanding the performance of binary classifiers. However, our case is a bit out of the ordinary. We have a very imbalanced dataset. Nonetheless, let’s have a look at our ROC curve:

1fpr, tpr, thresholds = roc_curve(error_df.true_class, error_df.reconstruction_error)
2roc_auc = auc(fpr, tpr)
3
4plt.title('Receiver Operating Characteristic')
5plt.plot(fpr, tpr, label='AUC = %0.4f'% roc_auc)
6plt.legend(loc='lower right')
7plt.plot([0,1],[0,1],'r--')
8plt.xlim([-0.001, 1])
9plt.ylim([0, 1.001])
10plt.ylabel('True Positive Rate')
11plt.xlabel('False Positive Rate')
12plt.show();

The ROC curve plots the true positive rate versus the false positive rate, over different threshold values. Basically, we want the blue line to be as close as possible to the upper left corner. While our results look pretty good, we have to keep in mind of the nature of our dataset. ROC doesn’t look very useful for us. Onward…

Precision vs Recall

source: Wikipedia

Precision and recall are defined as follows:

$\text{Precision} = \frac{\text{true positives}}{\text{true positives} + \text{false positives}}$

$\text{Recall} = \frac{\text{true positives}}{\text{true positives} + \text{false negatives}}$

Let’s take an example from Information Retrieval in order to better understand what precision and recall are. Precision measures the relevancy of obtained results. Recall, on the other hand, measures how many relevant results are returned. Both values can take values between 0 and 1. You would love to have a system with both values being equal to 1.

Let’s return to our example from Information Retrieval. High recall but low precision means many results, most of which has low or no relevancy. When precision is high but recall is low we have the opposite - few returned results with very high relevancy. Ideally, you would want high precision and high recall - many results with that are highly relevant.

1precision, recall, th = precision_recall_curve(error_df.true_class, error_df.reconstruction_error)
2plt.plot(recall, precision, 'b', label='Precision-Recall curve')
3plt.title('Recall vs Precision')
4plt.xlabel('Recall')
5plt.ylabel('Precision')
6plt.show()

A high area under the curve represents both high recall and high precision, where high precision relates to a low false positive rate, and high recall relates to a low false negative rate. High scores for both show that the classifier is returning accurate results (high precision), as well as returning a majority of all positive results (high recall).

1plt.plot(th, precision[1:], 'b', label='Threshold-Precision curve')
2plt.title('Precision for different threshold values')
3plt.xlabel('Threshold')
4plt.ylabel('Precision')
5plt.show()

You can see that as the reconstruction error increases our precision rises as well. Let’s have a look at the recall:

1plt.plot(th, recall[1:], 'b', label='Threshold-Recall curve')
2plt.title('Recall for different threshold values')
3plt.xlabel('Reconstruction error')
4plt.ylabel('Recall')
5plt.show()

Here, we have the exact opposite situation. As the reconstruction error increases the recall decreases.

Prediction

Our model is a bit different this time. It doesn’t know how to predict new values. But we don’t need that. In order to predict whether or not a new/unseen transaction is normal or fraudulent, we’ll calculate the reconstruction error from the transaction data itself. If the error is larger than a predefined threshold, we’ll mark it as a fraud (since our model should have a low error on normal transactions). Let’s pick that value:

1threshold = 2.9

And see how well we’re dividing the two types of transactions:

1groups = error_df.groupby('true_class')
2fig, ax = plt.subplots()
3
4for name, group in groups:
5    ax.plot(group.index, group.reconstruction_error, marker='o', ms=3.5, linestyle='',
6            label= "Fraud" if name == 1 else "Normal")
7ax.hlines(threshold, ax.get_xlim()[0], ax.get_xlim()[1], colors="r", zorder=100, label='Threshold')
8ax.legend()
9plt.title("Reconstruction error for different classes")
10plt.ylabel("Reconstruction error")
11plt.xlabel("Data point index")
12plt.show();

I know, that chart might be a bit deceiving. Let’s have a look at the confusion matrix:

1y_pred = [1 if e > threshold else 0 for e in error_df.reconstruction_error.values]
2conf_matrix = confusion_matrix(error_df.true_class, y_pred)
3
4plt.figure(figsize=(12, 12))
5sns.heatmap(conf_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, fmt="d");
6plt.title("Confusion matrix")
7plt.ylabel('True class')
8plt.xlabel('Predicted class')
9plt.show()

Our model seems to catch a lot of the fraudulent cases. Of course, there is a catch (see what I did there?). The number of normal transactions classified as frauds is really high. Is this really a problem? Probably it is. You might want to increase or decrease the value of the threshold, depending on the problem. That one is up to you.

Conclusion

We’ve created a very simple Deep Autoencoder in Keras that can reconstruct what non fraudulent transactions looks like. Initially, I was a bit skeptical about whether or not this whole thing is gonna work out, bit it kinda did. Think about it, we gave a lot of one-class examples (normal transactions) to a model and it learned (somewhat) how to discriminate whether or not new examples belong to that same class. Isn’t that cool? Our dataset was kind of magical, though. We really don’t know what the original features look like.

Keras gave us very clean and easy to use API to build a non-trivial Deep Autoencoder. You can search for TensorFlow implementations and see for yourself how much boilerplate you need in order to train one. Can you apply a similar model to a different problem?

The source code and pre-trained model are available on GitHub here.

References

• Building Autoencoders in Keras
• Stanford tutorial on Autoencoders
• Stacked Autoencoders in TensorFlow