2020/10/26

TensorFlow手寫數字辨識_CNN

TensorFlow手寫數字辨識_CNN

以 MLP 方式建立的模型,正確率約為 96%,要再進一步提升正確率,就要使用 Yann Lecun 提出的 CNN Convolutional Neural Network。

CNN 簡介

卷積運算就是將一個影像,經過卷積運算後,產生多個影像,分為兩個部分

  1. 卷積與縮減取樣,提取影像的特徵

    經過第一次卷積、第一次縮減取樣、第二次卷積、第二次縮減取樣,提取影像的特徵

  2. 完全連結神經網路

    提取影像特徵後,reshape 為1維的向量,送進 平坦層、隱藏層、輸出層 組成的累身經網路進行處理

池化層用來 downsampling,優點:

  1. 減少所需處理的資料點:減少後續運算所需時間
  2. 讓影像位置差異變小:手寫數字的位置不同,會影響辨識結果,減少影像大小可讓位置差異變小
  3. 參數的數量與計算量下降:控制 overfitting 的問題

tensorflow CNN

import tensorflow as tf
import numpy as np

# STEP 1 讀取資料
mnist = tf.keras.datasets.mnist
# Tuple of Numpy arrays: (x_train, y_train), (x_test, y_test)

(x_train, y_train), (x_test, y_test) = mnist.load_data()

# 將 training 的 input 資料 28*28 的 2維陣列 轉為 1維陣列,再轉成 float32
# 每一個圖片,都變成 784 個 float 的 array
# training 與 testing 資料數量分別是 60000 與 10000 筆
# X_train_2D 是 [60000, 28*28] 的 2維陣列
x_train_2D = x_train.reshape(60000, 28*28).astype('float32')
x_test_2D = x_test.reshape(10000, 28*28).astype('float32')
print('x_train_2D.shape=', x_train_2D.shape)
# x_train_2D.shape=(60000, 784)

# 將圖片的數字 (0~255) 標準化,最簡單的方法就是直接除以 255
# x_train_norm 是標準化後的結果,每一個數字介於 0~1 之間
x_train_norm = x_train_2D/255
x_test_norm = x_test_2D/255

# 將 training 的 label 進行 one-hot encoding,例如數字 7 經過 One-hot encoding 轉換後是 array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0.], dtype=float32),即第7個值為 1

y_train_one_hot_tf=tf.one_hot(y_train,10)
y_test_one_hot_tf=tf.one_hot(y_test,10)

y_train_one_hot = None
y_test_one_hot = None
with tf.compat.v1.Session() as sess:
    init = tf.compat.v1.global_variables_initializer()
    sess.run(init)
    y_train_one_hot = sess.run(y_train_one_hot_tf)
    y_test_one_hot = sess.run(y_test_one_hot_tf)

# 將 x_train, y_train 分成 train 與 validation 兩個部分
x_train_norm_data = x_train_norm[0:50000]
x_train_norm_validation = x_train_norm[50000:60000]

y_train_one_hot_data = y_train_one_hot[0:50000]
y_train_one_hot_validation = y_train_one_hot[50000:60000]


### 建立模型

# 先建立一些共用的函數
def weight(shape):
    return tf.Variable(tf.random.truncated_normal(shape, stddev=0.1),
                       name ='W')
# bias 張量,先以 constant 建立常數,然後用 Variable 建立張量變數
def bias(shape):
    return tf.Variable(tf.constant(0.1, shape=shape)
                       , name = 'b')
# 卷積運算 功能相當於濾鏡
#  x 是輸入的影像,必須是 4 維的張量
#  W 是 filter weight 濾鏡的權重,後續以隨機方式產生 filter weight
#  strides 是 濾鏡的跨步 step,設定為 [1,1,1,1],格式是 [1, stride, stride, 1],濾鏡每次移動時,從左到右,上到下,各移動 1 步
#  padding 是 'SAME',此模式會在邊界以外 補0 再做運算,讓輸入與輸出影像為相同大小
def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1,1,1,1],
                        padding='SAME')

# 建立池化層,進行影像的縮減取樣
#  x 是輸入的影像,必須是 4 維的張量
#  ksize 是縮減取樣窗口的大小,設定為 [1,2,2,1],格式為 [1, height, width, 1],也就是高度 2 寬度 2 的窗口
#  stides 是縮減取樣窗口的跨步 step,設定為 [1,2,2,1],格式為 [1, stride, stride, 1],也就是縮減取樣窗口,由左到右,由上到下,各2步
#  原本 28x28 的影像,經過 max-pool 後,會縮小為 14x14
def max_pool_2x2(x):
    return tf.nn.max_pool2d(x, ksize=[1,2,2,1],
                          strides=[1,2,2,1],
                          padding='SAME')


# 輸入層
with tf.name_scope('Input_Layer'):
    # placeholder 會傳入影像
    x = tf.compat.v1.placeholder("float",shape=[None, 784],name="x")
    # x 原本為 1 維張量,要 reshape 為 4 維張量
    # 第 1 維 -1,因為後續訓練要透過 placeholder 輸入的資料筆數不固定
    # 第 2, 3 維,是 28, 28,因為影像為 28x28
    # 第 4 維是 1,因為是單色的影像,就設定為 1,如果是彩色,要設定為 3 (RGB)
    x_image = tf.reshape(x, [-1, 28, 28, 1])

# CNN Layer 1
# 用來提取特徵,卷積運算後,會產生 16 個影像,大小仍為 28x28
with tf.name_scope('C1_Conv'):
    # filter weight 大小為 5x5
    # 因為是單色,第 3 維設定為 1
    # 要產生 16 個影像,所以第 4 維設定為 16
    W1 = weight([5,5,1,16])

    # 因為產生 16 個影像,所以輸入餐次 shape = 16
    b1 = bias([16])

    # 卷積運算
    Conv1=conv2d(x_image, W1)+ b1
    # ReLU 激活函數
    C1_Conv = tf.nn.relu(Conv1 )

# 池化層用來 downsampling,將影像由 28x28 縮小為 14x14,影像數量仍為 16
with tf.name_scope('C1_Pool'):
    C1_Pool = max_pool_2x2(C1_Conv)

# CNN Layer 2
# 第二次卷積運算,將 16 個影像轉換為 36 個影像,卷積運算不改變影像大小,仍為 14x14
with tf.name_scope('C2_Conv'):
    # filter weight 大小為 5x5
    # 第 3 維是 16,因為卷積層1 的影像數量為 16
    # 第 4 維設定為 36,因為將 16 個影像轉換為 36個
    W2 = weight([5,5,16,36])
    # 因為產生 36 個影像,所以輸入餐次 shape = 36
    b2 = bias([36])
    Conv2=conv2d(C1_Pool, W2)+ b2
    # relu 會將負數的點轉換為 0
    C2_Conv = tf.nn.relu(Conv2)

# 池化層2用來 downsampling,將影像由 14x14 縮小為 7x7,影像數量仍為 36
with tf.name_scope('C2_Pool'):
    C2_Pool = max_pool_2x2(C2_Conv)

# Fully Connected Layer
# 平坦層,將 36個 7x7 影像,轉換為 1 維向量,長度為 36x7x7= 1764,也就是 1764 個 float,作為輸入資料
with tf.name_scope('D_Flat'):
    D_Flat = tf.reshape(C2_Pool, [-1, 1764])

with tf.name_scope('D_Hidden_Layer'):
    W3= weight([1764, 128])
    b3= bias([128])
    D_Hidden = tf.nn.relu(
                  tf.matmul(D_Flat, W3)+b3)

    ## Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
    # D_Hidden_Dropout= tf.nn.dropout(D_Hidden, keep_prob=0.8)
    D_Hidden_Dropout= tf.nn.dropout(D_Hidden, rate = 0.2)

# 輸出層, 10 個神經元
#  y_predict = softmax(D_Hidden_Dropout * W4 + b4)
with tf.name_scope('Output_Layer'):
    # 因為上一層 D_Hidden 是 128 個神經元,所以第1維是 128
    W4 = weight([128,10])
    b4 = bias([10])
    y_predict= tf.nn.softmax(
                 tf.matmul(D_Hidden_Dropout, W4)+b4)


### 設定訓練模型最佳化步驟
# 使用反向傳播演算法,訓練多層感知模型
with tf.name_scope("optimizer"):

    y_label = tf.compat.v1.placeholder("float", shape=[None, 10],
                              name="y_label")

    loss_function = tf.reduce_mean(
                      tf.nn.softmax_cross_entropy_with_logits
                         (logits=y_predict ,
                          labels=y_label))

    optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=0.0001) \
                    .minimize(loss_function)


### 設定評估模型
with tf.name_scope("evaluate_model"):
    correct_prediction = tf.equal(tf.argmax(y_predict, 1),
                                  tf.argmax(y_label, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))


### 訓練模型

trainEpochs = 30
batchSize = 100
totalBatchs = int(len(x_train_norm_data)/batchSize)
epoch_list=[];accuracy_list=[];loss_list=[];
from time import time

with tf.compat.v1.Session() as sess:
    startTime=time()

    sess.run(tf.compat.v1.global_variables_initializer())

    for epoch in range(trainEpochs):
        for i in range(totalBatchs):
            # batch_x, batch_y = mnist.train.next_batch(batchSize)
            batch_x = x_train_norm_data[i*batchSize:(i+1)*batchSize]
            batch_y = y_train_one_hot_data[i*batchSize:(i+1)*batchSize]

            sess.run(optimizer,feed_dict={x: batch_x,
                                          y_label: batch_y})

        loss,acc = sess.run([loss_function,accuracy],
                            feed_dict={x: x_train_norm_validation,
                                       y_label: y_train_one_hot_validation})

        epoch_list.append(epoch)
        loss_list.append(loss)
        accuracy_list.append(acc)

        print("Train Epoch:", '%02d' % (epoch+1), "Loss=","{:.9f}".format(loss)," Accuracy=",acc)

    duration =time()-startTime
    print("Train Finished takes:",duration)

    ## 評估模型準確率
    print("Accuracy:",
      sess.run(accuracy,feed_dict={x: x_test_norm,
                                   y_label:y_test_one_hot}))
    # 前 5000 筆
    print("Accuracy:",
      sess.run(accuracy,feed_dict={x: x_test_norm[:5000],
                                   y_label: y_test_one_hot[:5000]}))
    # 後 5000 筆
    print("Accuracy:",
      sess.run(accuracy,feed_dict={x: x_test_norm[5000:],
                                   y_label: y_test_one_hot[5000:]}))

    ## 預測機率
    y_predict=sess.run(y_predict,
                   feed_dict={x: x_test_norm[:5000]})

    ## 預測結果
    prediction_result=sess.run(tf.argmax(y_predict,1),
                           feed_dict={x: x_test_norm ,
                                      y_label: y_test_one_hot})

    ## 儲存模型
    saver = tf.train.Saver()
    save_path = saver.save(sess, "saveModel/CNN_model1")
    print("Model saved in file: %s" % save_path)
    merged = tf.summary.merge_all()
    # 可將 計算圖,透過 TensorBoard 視覺化
    train_writer = tf.summary.FileWriter('log/CNN',sess.graph)


# matplotlib 列印 loss, accuracy 折線圖
import matplotlib.pyplot as plt

fig = plt.gcf()
# fig.set_size_inches(4,2)
plt.plot(epoch_list, loss_list, label = 'loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['loss'], loc='upper left')
plt.savefig('loss.png')


fig = plt.gcf()
# fig.set_size_inches(4,2)
plt.plot(epoch_list, accuracy_list,label="accuracy" )

plt.ylim(0.8,1)
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['accuracy'], loc='upper right')
plt.savefig('accuracy.png')

############
# 查看多筆資料,以及 label
import matplotlib.pyplot as plt
def plot_images_labels_prediction(images,labels,prediction,idx,filename, num=10):
    fig = plt.gcf()
    fig.set_size_inches(12, 14)
    if num>25: num=25
    for i in range(0, num):
        ax=plt.subplot(5,5, 1+i)

        # 將 images 的 784 個數字轉換為 28x28
        ax.imshow(np.reshape(images[idx],(28, 28)), cmap='binary')

        # 轉換 one_hot label 為數字
        title= "label=" +str(np.argmax(labels[idx]))
        if len(prediction)>0:
            title+=",predict="+str(prediction[idx])

        ax.set_title(title,fontsize=10)
        ax.set_xticks([]);ax.set_yticks([])
        idx+=1
    plt.savefig(filename)


plot_images_labels_prediction(x_test_norm,
                              y_test_one_hot,
                              prediction_result,0, "result.png", num=10)

# 找出預測錯誤
for i in range(400):
    if prediction_result[i]!=np.argmax(y_test_one_hot[i]):
        print("i="+str(i)+
              "   label=",np.argmax(y_test_one_hot[i]),
              "predict=",prediction_result[i])
Train Epoch: 01 Loss= 1.604377151  Accuracy= 0.8872
Train Epoch: 02 Loss= 1.547111511  Accuracy= 0.9281
Train Epoch: 03 Loss= 1.525221825  Accuracy= 0.9447
Train Epoch: 04 Loss= 1.516423583  Accuracy= 0.9511
Train Epoch: 05 Loss= 1.507740974  Accuracy= 0.9584
Train Epoch: 06 Loss= 1.503444791  Accuracy= 0.9636
Train Epoch: 07 Loss= 1.496760130  Accuracy= 0.9683
Train Epoch: 08 Loss= 1.494633555  Accuracy= 0.9712
Train Epoch: 09 Loss= 1.492025375  Accuracy= 0.9724
Train Epoch: 10 Loss= 1.491448402  Accuracy= 0.9735
Train Epoch: 11 Loss= 1.488568783  Accuracy= 0.9751
Train Epoch: 12 Loss= 1.488826513  Accuracy= 0.9745
Train Epoch: 13 Loss= 1.485750437  Accuracy= 0.9778
Train Epoch: 14 Loss= 1.484605789  Accuracy= 0.9798
Train Epoch: 15 Loss= 1.483879209  Accuracy= 0.9788
Train Epoch: 16 Loss= 1.482506037  Accuracy= 0.9808
Train Epoch: 17 Loss= 1.482969046  Accuracy= 0.9796
Train Epoch: 18 Loss= 1.481315017  Accuracy= 0.9811
Train Epoch: 19 Loss= 1.480247617  Accuracy= 0.983
Train Epoch: 20 Loss= 1.480669379  Accuracy= 0.9817
Train Epoch: 21 Loss= 1.480412602  Accuracy= 0.9824
Train Epoch: 22 Loss= 1.479805708  Accuracy= 0.983
Train Epoch: 23 Loss= 1.479858279  Accuracy= 0.9827
Train Epoch: 24 Loss= 1.479218960  Accuracy= 0.9834
Train Epoch: 25 Loss= 1.479144573  Accuracy= 0.9829
Train Epoch: 26 Loss= 1.478820801  Accuracy= 0.9838
Train Epoch: 27 Loss= 1.477338433  Accuracy= 0.9857
Train Epoch: 28 Loss= 1.478171706  Accuracy= 0.9847
Train Epoch: 29 Loss= 1.477008104  Accuracy= 0.9856
Train Epoch: 30 Loss= 1.477438688  Accuracy= 0.9845
Train Finished takes: 1763.7836382389069
Accuracy: 0.988
Accuracy: 0.9814
Accuracy: 0.9928

i=18   label= 3 predict= 5
i=290   label= 8 predict= 4
i=321   label= 2 predict= 7
i=359   label= 9 predict= 8

2020/10/19

Keras 手寫阿拉伯數字辨識 CNN

Keras 手寫阿拉伯數字辨識 CNN

卷積神經網路 CNN 是由 Yann LeCun 提出,以下是以 CNN 進行 mnist 資料辨識。

MLP 與 CNN 的差異是 CNN 增加了卷積層1、池化層1、卷積層2、池化層2 的處理,用以提取特徵。

CNN 可分為兩個部分

  1. 影像的特徵擷取

    透過卷積層1、池化層1、卷積層2、池化層2 的處理,用以提取特徵

  2. 完全連結神經網路

    包含平坦層、隱藏層、輸出層組成的類神經網路

卷積運算的效果很類似濾鏡,擷取了不同的特徵。


卷積運算

  1. 以隨機方式產生, filter weight 大小為 3x3
  2. 將要轉換的影像,由左至右、上至下,依序選取 3x3 的矩陣
  3. 影像選取的矩陣 3x3,及 filter weight 3x3 乘積的結果,算出第一列第一行的數字
  1. 依照同樣的方式,完成所有運算

  2. 使用單一 filter weight 卷積運算產生影像

    以下是數字 7 (28x28) 的影像,使用隨機產生的 5x5 filter weight (w) 濾鏡,進行卷積運算後的結果。卷積運算不會改變圖片的大小,但運算後的結果,可提取輸入的不同特徵,ex: 邊緣、線條、角

  3. 使用多個 filter weight 卷積運算產生多個影像

    接下來隨機產生 16 個 filter weight,也就是 16 個濾鏡

    透過卷積運算,使用 16 個 filter weight,產生 16個影像,每一種影像提取了不同的特徵

  4. Max-Pool 運算

    可將影像縮減取樣 (downsampling),例如原本是 4x4 的影像,Max-Pool 後得到的影像為 2x2

  5. 使用 Max-Pool 轉換手寫數字影像

    將 16 個 28x28 影像,縮小為 16 個 14x14 的影像,影像數量不變

    Max-Pool downsampling 會縮小圖片,其優點是

    • 減少需要處理的資料點:減少運算時間
    • 讓影像位置差異變小:例如 7,在圖片中的位置不固定,可能偏某一側,但位置不同會影響辨識,減少影像大小,讓數字的位置差異變小
    • 參數的數量與計算量下降:降低發生 overfitting 的狀況

MNIST CNN

步驟

  1. 資料預處理 Preprocess:處理後產生 features (影像特徵值)與 label (數字的真實值)

  2. 建立模型:建立 CNN model

  3. 訓練模型:輸入 features, label,執行 10 次訓練週期

  4. 評估模型準確率:使用測試資料評估模型準確率

  5. 預測:利用 model,輸入測試資料進行預測

from keras.datasets import mnist
from keras.utils import np_utils
import numpy as np
np.random.seed(10)

## step 1 資料預處理 Preprocess:處理後產生 features (影像特徵值)與 label (數字的真實值)
# 讀取 mnist 資料
(x_Train, y_Train), (x_Test, y_Test) = mnist.load_data()
# 將 features (影像特徵值),轉換為 4 維矩陣
# 將 features,以 reshape 轉為 6000 x 28 x 28 x 1 的 4 維矩陣
x_Train4D = x_Train.reshape(x_Train.shape[0],28,28,1).astype('float32')
x_Test4D = x_Test.reshape(x_Test.shape[0],28,28,1).astype('float32')

# 將 features 標準化
x_Train4D_normalize = x_Train4D / 255
x_Test4D_normalize = x_Test4D / 255

# 以 Onehot Encoding 轉換 label
y_TrainOneHot = np_utils.to_categorical(y_Train)
y_TestOneHot = np_utils.to_categorical(y_Test)

#########
## step 2 建立模型:建立 CNN model
from keras.models import Sequential
from keras.layers import Dense,Dropout,Flatten,Conv2D,MaxPooling2D

# 線性堆疊模型
model = Sequential()

# 建立卷積層1
# 輸入數字影像 28x28 的大小,執行一次卷積運算,產生 16 個影像,卷積運算不會改變影像大小,結果還是 28x28
# filters=16             建立 16 個 filter weight
# kernel_size=(5,5)      每一個濾鏡大小為 5x5
# padding='same'         讓卷積運算產生的影像大小不變
# input_shape=(28,28,1)  第1, 2 維,是輸入的影像形狀 28x28,第 3 維,因為是單色灰階影像,所以是 1
# activation='relu'      設定 ReLU 激活函數
model.add(Conv2D(filters=16,
                 kernel_size=(5,5),
                 padding='same',
                 input_shape=(28,28,1),
                 activation='relu'))

# 建立池化層
# 輸入參數 pool_size=(2, 2),執行第一次縮減取樣,將 16 個 28x28 影像,縮小為 16 個 14x14 的影像
model.add(MaxPooling2D(pool_size=(2, 2)))

# 建立卷積層2
# 執行第二次卷積運算,將原本的 16 個影像,轉換為 36 個影像,卷積運算不會改變影像大小,結果還是 14x14
model.add(Conv2D(filters=36,
                 kernel_size=(5,5),
                 padding='same',
                 activation='relu'))

# 建立池化層2,並加入Dropout 避免 overfitting
# 執行第二次縮減取樣,將 36 個 14x14 的影像,縮小為 36 個 7x7 的影像
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

# 建立神經網路 (平坦層, 隱藏層, 輸出層)
# 建立平坦層
# 根據池化層2 的結果,共36 個 7x7 影像,轉換為 1維向量,長度是 36x7x7=1764,也就是 1764 個 float,正好對應到 1764 個神經元
model.add(Flatten())
# 建立隱藏層,共有 128 個神經元
model.add(Dense(128, activation='relu'))
# 加入 Dropout(0.5)
# 每次訓練迭代時,會隨機在神經網路中,放棄 50% 的神經元,以避免 overfitting
model.add(Dropout(0.5))
# 建立輸出層
# 共 10 個神經元,對應 0~9 共 10 個數字,並使用 softmax 激活函數進行轉換
# softmax 可將神經元的輸出,轉換為預測每一個數字的機率
model.add(Dense(10,activation='softmax'))

print(model.summary())

#######
## 訓練模型:輸入 features, label,執行 10 次訓練週期
model.compile(loss='categorical_crossentropy',
              optimizer='adam',metrics=['accuracy'])

# validation_split=0.2   80% 為訓練資料, 20% 驗證資料
# batch_size=300         每一批次300 筆資料
# verbose=2              顯示訓練過程
train_history=model.fit(x=x_Train4D_normalize,
                        y=y_TrainOneHot,validation_split=0.2,
                        epochs=20, batch_size=300,verbose=2)


# 模型訓練結果 結構存檔
from keras.models import model_from_json
json_string = model.to_json()
with open("model.config", "w") as text_file:
    text_file.write(json_string)

# 模型訓練結果 權重存檔
model.save_weights("model.weight")


import matplotlib.pyplot as plt
def save_train_history(train_acc,test_acc, filename):
    plt.clf()
    plt.plot(train_history.history[train_acc])
    plt.plot(train_history.history[test_acc])
    plt.title('Train History')
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig(filename)


save_train_history('accuracy','val_accuracy', 'acc.png')

save_train_history('loss','val_loss', 'loss.png')


#####
# step 4 評估模型準確率:使用測試資料評估模型準確率

scores = model.evaluate(x_Test4D_normalize , y_TestOneHot)
scores[1]

#####
# step 5 預測:利用 model,輸入測試資料進行預測
prediction=model.predict_classes(x_Test4D_normalize)
prediction[:10]

# 查看預測結果
import matplotlib.pyplot as plt
def plot_images_labels_prediction(images,labels,prediction,filename, idx, num=10):
    fig = plt.gcf()
    fig.set_size_inches(12, 14)
    if num>25: num=25
    for i in range(0, num):
        ax=plt.subplot(5,5, 1+i)
        ax.imshow(images[idx], cmap='binary')

        ax.set_title("label=" +str(labels[idx])+
                     ",predict="+str(prediction[idx])
                     ,fontsize=10)

        ax.set_xticks([]);ax.set_yticks([])
        idx+=1
    plt.savefig(filename)

plot_images_labels_prediction(x_Test,y_Test,prediction, 'predict.png', idx=0)

####
# confusion maxtrix

import pandas as pd
crosstab1 = pd.crosstab(y_Test,prediction,
            rownames=['label'],colnames=['predict'])

print()
print(crosstab1)

df = pd.DataFrame({'label':y_Test, 'predict':prediction})

df[(df.label==5)&(df.predict==3)]

列印 model

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 28, 28, 16)        416
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 16)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 36)        14436
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 36)          0
_________________________________________________________________
dropout_1 (Dropout)          (None, 7, 7, 36)          0
_________________________________________________________________
flatten_1 (Flatten)          (None, 1764)              0
_________________________________________________________________
dense_1 (Dense)              (None, 128)               225920
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_2 (Dense)              (None, 10)                1290
=================================================================
Total params: 242,062
Trainable params: 242,062
Non-trainable params: 0

訓練過程,可發現 loss 越來越小,accuracy 越來越高

Train on 48000 samples, validate on 12000 samples
Epoch 1/20
 - 58s - loss: 0.4736 - accuracy: 0.8517 - val_loss: 0.1006 - val_accuracy: 0.9694
Epoch 2/20
 - 63s - loss: 0.1326 - accuracy: 0.9604 - val_loss: 0.0652 - val_accuracy: 0.9813
Epoch 3/20
 - 65s - loss: 0.0980 - accuracy: 0.9700 - val_loss: 0.0555 - val_accuracy: 0.9838
Epoch 4/20
 - 67s - loss: 0.0791 - accuracy: 0.9761 - val_loss: 0.0479 - val_accuracy: 0.9862
Epoch 5/20
 - 61s - loss: 0.0698 - accuracy: 0.9779 - val_loss: 0.0442 - val_accuracy: 0.9873
Epoch 6/20
 - 59s - loss: 0.0616 - accuracy: 0.9813 - val_loss: 0.0434 - val_accuracy: 0.9875
Epoch 7/20
 - 62s - loss: 0.0531 - accuracy: 0.9835 - val_loss: 0.0370 - val_accuracy: 0.9893
Epoch 8/20
 - 63s - loss: 0.0496 - accuracy: 0.9843 - val_loss: 0.0363 - val_accuracy: 0.9904
Epoch 9/20
 - 61s - loss: 0.0455 - accuracy: 0.9863 - val_loss: 0.0347 - val_accuracy: 0.9908
Epoch 10/20
 - 65s - loss: 0.0417 - accuracy: 0.9870 - val_loss: 0.0319 - val_accuracy: 0.9920
Epoch 11/20
 - 69s - loss: 0.0375 - accuracy: 0.9880 - val_loss: 0.0309 - val_accuracy: 0.9912
Epoch 12/20
 - 62s - loss: 0.0357 - accuracy: 0.9891 - val_loss: 0.0341 - val_accuracy: 0.9907
Epoch 13/20
 - 73s - loss: 0.0347 - accuracy: 0.9894 - val_loss: 0.0332 - val_accuracy: 0.9909
Epoch 14/20
 - 64s - loss: 0.0314 - accuracy: 0.9902 - val_loss: 0.0312 - val_accuracy: 0.9921
Epoch 15/20
 - 65s - loss: 0.0298 - accuracy: 0.9907 - val_loss: 0.0296 - val_accuracy: 0.9923
Epoch 16/20
 - 66s - loss: 0.0260 - accuracy: 0.9914 - val_loss: 0.0312 - val_accuracy: 0.9920
Epoch 17/20
 - 69s - loss: 0.0255 - accuracy: 0.9923 - val_loss: 0.0270 - val_accuracy: 0.9933
Epoch 18/20
 - 67s - loss: 0.0243 - accuracy: 0.9924 - val_loss: 0.0305 - val_accuracy: 0.9921
Epoch 19/20
 - 62s - loss: 0.0241 - accuracy: 0.9922 - val_loss: 0.0299 - val_accuracy: 0.9925
Epoch 20/20
 - 71s - loss: 0.0214 - accuracy: 0.9933 - val_loss: 0.0311 - val_accuracy: 0.9918

訓練與驗證的準確率都越來越高,誤差越來越低,且沒有 overfitting 的現象

預測的 scores,準確率 0.9926

[0.021972040887850517, 0.9926000237464905]

這是前 10 筆預測資料

crosstab 結果

predict    0     1     2    3    4    5    6     7    8    9
label
0        976     1     0    0    0    0    1     1    1    0
1          0  1133     1    0    0    0    0     1    0    0
2          2     0  1024    0    0    0    0     4    2    0
3          0     0     1  999    0    3    0     2    3    2
4          0     0     0    0  978    0    1     0    0    3
5          1     0     0    4    0  883    3     0    0    1
6          3     2     0    0    2    1  949     0    1    0
7          0     2     3    0    0    0    0  1022    1    0
8          3     1     1    1    0    0    0     0  967    1
9          1     0     0    0    6    2    0     4    1  995

References

TensorFlow+Keras深度學習人工智慧實務應用

何時使用MLP,CNN和RNN神經網絡

2020/10/12

keras cifar-10

cifar-10 是由 Alex Krizhevsky, Vinod Nair, Geoffery Hinton 收集的一個用於影像辨識資料集,共10類圖片:飛機、汽車、鳥、貓、鹿、狗、青蛙、馬、船、卡車。跟 MNIST 將比, cifar-10 的資料是彩色,雜訊較多,大小不一,角度不同,顏色不同。所以難度比 MNIST 高。

cifar-10 共有 60000 個 32x32 彩色圖像,每一類有 6000 個,共有 50000個訓練圖像及 10000 個測試圖像。

cifar-10 資料集

訓練資料由 images, labels 組成,ylabeltrain 是圖形資料的真實值,每一個數字代表一類圖片

0: airplain, 1: automobile, 2: bird, 3: cat, 4: deer, 5: dog, 6: frog, 7: horse, 8: ship, 9: truck

ximgtrain 的維度如下:有 50000 筆,影像大小為 32x32,第四維因為每一個像素點是 RGB 三原色組成,數值範圍是 0~255,所以是 3

x_img_train.shape: (50000, 32, 32, 3)
import numpy
from keras.datasets import cifar10
import numpy as np
np.random.seed(10)

###########
# 資料準備,載入 cifar10
# 資料會放在 ~/.keras/datasets/cifar-10-batches-py
(x_img_train,y_label_train), (x_img_test, y_label_test)=cifar10.load_data()

# print('train:',len(x_img_train), ', x_img_train.shape:',x_img_train.shape, ', y_label_train:', y_label_train.shape)
# print('test :',len(x_img_test), ', x_img_test.shape:', x_img_test.shape, ', y_label_test:', y_label_test.shape)

## train: 50000 , x_img_train.shape: (50000, 32, 32, 3) , y_label_train: (50000, 1)
## test : 10000 , x_img_test.shape: (10000, 32, 32, 3) , y_label_test: (10000, 1)
# print('x_img_test[0]:', x_img_test[0])

###########
# 查看多筆資料與 label

# 定義 label_dict
label_dict={0:"airplane",1:"automobile",2:"bird",3:"cat",4:"deer",
            5:"dog",6:"frog",7:"horse",8:"ship",9:"truck"}

# 產生圖片, label, prediction 的 preview
import matplotlib.pyplot as plt
def plot_images_labels_prediction(images,labels,prediction,idx,filename,num=10):
    fig = plt.gcf()
    fig.set_size_inches(12, 14)
    if num>25: num=25
    for i in range(0, num):
        ax=plt.subplot(5,5, 1+i)
        ax.imshow(images[idx],cmap='binary')

        title=str(i)+','+label_dict[labels[i][0]]
        if len(prediction)>0:
            title+='=>'+label_dict[prediction[i]]

        ax.set_title(title,fontsize=10)
        ax.set_xticks([]);ax.set_yticks([])
        idx+=1
    plt.savefig(filename)

# 查看前 10 筆資料
# plot_images_labels_prediction(x_img_train,y_label_train,[],0, 'x_img_train_0_10.png', num=10)


###########
# 對圖片進行預處理
# image normalize

# 查看圖片的第一個點
# print('x_img_train[0][0][0]=', x_img_train[0][0][0])
## x_img_train[0][0][0]= [59 62 63]

# normalize 標準化,可提升模型的準確度
x_img_train_normalize = x_img_train.astype('float32') / 255.0
x_img_test_normalize = x_img_test.astype('float32') / 255.0

# print('x_img_train_normalize[0][0][0]=', x_img_train_normalize[0][0][0])
## x_img_train_normalize[0][0][0]= [0.23137255 0.24313726 0.24705882]

## 將 label 轉為 one hot encoding
from keras.utils import np_utils
y_label_train_OneHot = np_utils.to_categorical(y_label_train)
y_label_test_OneHot = np_utils.to_categorical(y_label_test)

# print('y_label_train[:5]=', y_label_train[:5])
# print('y_label_train_OneHot.shape=', y_label_train_OneHot.shape)
# print('y_label_train_OneHot[:5]', y_label_train_OneHot[:5])
####
# y_label_train[:5]= [[6]
#  [9]
#  [9]
#  [4]
#  [1]]
# y_label_train_OneHot.shape= (50000, 10)
# y_label_train_OneHot[:5] [[0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]
#  [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
#  [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
#  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
#  [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]

cifar-10 CNN

模型對應的程式

列印 model.summary()

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 32, 32, 32)        896
_________________________________________________________________
dropout_1 (Dropout)          (None, 32, 32, 32)        0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 16, 16, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 16, 16, 64)        18496
_________________________________________________________________
dropout_2 (Dropout)          (None, 16, 16, 64)        0
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 64)          0
_________________________________________________________________
flatten_1 (Flatten)          (None, 4096)              0
_________________________________________________________________
dropout_3 (Dropout)          (None, 4096)              0
_________________________________________________________________
dense_1 (Dense)              (None, 1024)              4195328
_________________________________________________________________
dropout_4 (Dropout)          (None, 1024)              0
_________________________________________________________________
dense_2 (Dense)              (None, 10)                10250
=================================================================
Total params: 4,224,970
Trainable params: 4,224,970
Non-trainable params: 0
import numpy
from keras.datasets import cifar10
import numpy as np
np.random.seed(10)

###########
# 資料準備,載入 cifar10
# 資料會放在 ~/.keras/datasets/cifar-10-batches-py
(x_img_train,y_label_train), (x_img_test, y_label_test)=cifar10.load_data()

###########
# 對圖片進行預處理

# normalize 標準化,可提升模型的準確度
x_img_train_normalize = x_img_train.astype('float32') / 255.0
x_img_test_normalize = x_img_test.astype('float32') / 255.0

## 將 label 轉為 one hot encoding
from keras.utils import np_utils
y_label_train_OneHot = np_utils.to_categorical(y_label_train)
y_label_test_OneHot = np_utils.to_categorical(y_label_test)

#########
# 建立模型

from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D

model = Sequential()

#卷積層 1與池化層1
## 輸入影像為 32x32,會產生 32 個影像,結果仍是 32x32
## filters=32 是隨機產生 32 個濾鏡 filter weight
## kernel_size=(3,3) 濾鏡大小為 3x3
## padding='same'  讓卷積運算的結果產生的影像大小不變
## activation='relu'  設定 ReLU activation function
model.add(Conv2D(filters=32,kernel_size=(3,3),
                 input_shape=(32, 32,3),
                 activation='relu',
                 padding='same'))
## 加入 Dropout 避免 overfitting
## 0.25 是每一次訓練迭代時,會隨機丟棄 25% 神經元
model.add(Dropout(rate=0.25))
## 池化層1
## pool_size 是縮減取樣,會縮小為 16x16,仍為 32 個
model.add(MaxPooling2D(pool_size=(2, 2)))

#卷積層2與池化層2
## 將 32 個影像,轉換為 64 個
model.add(Conv2D(filters=64, kernel_size=(3, 3),
                 activation='relu', padding='same'))
model.add(Dropout(0.25))
## 縮小影像,結果為 8x8 共 64 個影像
model.add(MaxPooling2D(pool_size=(2, 2)))

#Step3  建立神經網路(平坦層、隱藏層、輸出層)
## 將 64 個 8x8 影像轉換為 1 維,64*8*8=4096 個 float
model.add(Flatten())
## 加入 Dropout,隨機丟棄 25%
model.add(Dropout(rate=0.25))

## 建立隱藏層,共 1024 個神經元
model.add(Dense(1024, activation='relu'))
model.add(Dropout(rate=0.25))

## 輸出層
model.add(Dense(10, activation='softmax'))

# print(model.summary())

####################
import matplotlib.pyplot as plt
def show_train_history(train_acc,test_acc, filename):
    plt.gcf()
    plt.plot(train_history.history[train_acc])
    plt.plot(train_history.history[test_acc])
    plt.title('Train History')
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig(filename)

#### 判斷是否可載入已經訓練好的模型
try:
    model.load_weights("SaveModel/cifarCnnModelnew.h5")
    print("載入模型成功!繼續訓練模型")
except :
    print("載入模型失敗!開始訓練一個新模型")


#### 進行訓練

model.compile(loss='categorical_crossentropy',
              optimizer='adam', metrics=['accuracy'])

train_history=model.fit(x_img_train_normalize, y_label_train_OneHot,
                        validation_split=0.2,
                        epochs=10, batch_size=128, verbose=1)

show_train_history('accuracy','val_accuracy', 'accuracy.png')
show_train_history('loss','val_loss', 'loss.png')

#######
# 評估模型準確率
scores = model.evaluate(x_img_test_normalize,
                        y_label_test_OneHot, verbose=0)
print("scores[1]=", scores[1])

## 進行預測
prediction=model.predict_classes(x_img_test_normalize)

###########
# 查看多筆資料與 label

# 定義 label_dict
label_dict={0:"airplane",1:"automobile",2:"bird",3:"cat",4:"deer",
            5:"dog",6:"frog",7:"horse",8:"ship",9:"truck"}

# 產生圖片, label, prediction 的 preview
import matplotlib.pyplot as plt
def plot_images_labels_prediction(images,labels,prediction,idx,filename,num=10):
    fig = plt.gcf()
    fig.set_size_inches(12, 14)
    if num>25: num=25
    for i in range(0, num):
        ax=plt.subplot(5,5, 1+i)
        ax.imshow(images[idx],cmap='binary')

        title=str(i)+','+label_dict[labels[i][0]]
        if len(prediction)>0:
            title+='=>'+label_dict[prediction[i]]

        ax.set_title(title,fontsize=10)
        ax.set_xticks([]);ax.set_yticks([])
        idx+=1
    plt.savefig(filename)

## 列印前 10 筆預測結果
plot_images_labels_prediction(x_img_test,y_label_test,prediction,0,'prediction.png', num=10)


# 查看預測機率
Predicted_Probability=model.predict(x_img_test_normalize)

# y: 真實值
# prediciton: 預測結果
# x_img: 預測的影像
# Predicted_Probability: 預測機率
# i: 資料 index
def show_Predicted_Probability(y,prediction,
                               x_img,Predicted_Probability,i):
    print('-------------------')
    print('label:',label_dict[y[i][0]],
          'predict:',label_dict[prediction[i]])
    fig = plt.gcf()
    plt.figure(figsize=(2,2))
    plt.imshow(np.reshape(x_img_test[i],(32, 32,3)))
    plt.savefig(""+str(i)+".png")
    for j in range(10):
        print(label_dict[j]+
              ' Probability:%1.9f'%(Predicted_Probability[i][j]))

show_Predicted_Probability(y_label_test,prediction,
                           x_img_test,Predicted_Probability,0)

# label: cat predict: cat
# airplane Probability:0.000472784
# automobile Probability:0.001096419
# bird Probability:0.008890972
# cat Probability:0.852500975
# deer Probability:0.010386771
# dog Probability:0.074663654
# frog Probability:0.035179924
# horse Probability:0.002779935
# ship Probability:0.010328157
# truck Probability:0.003700291

show_Predicted_Probability(y_label_test,prediction,
                           x_img_test,Predicted_Probability,3)

# label: airplane predict: airplane
# airplane Probability:0.616022110
# automobile Probability:0.032570492
# bird Probability:0.073217131
# cat Probability:0.006363209
# deer Probability:0.030436775
# dog Probability:0.001208493
# frog Probability:0.001075586
# horse Probability:0.001057812
# ship Probability:0.235320851
# truck Probability:0.002727570

#####
# confusion matrix

print("prediction.shape=", str(prediction.shape), ", y_label_test.shape=",str(y_label_test.shape))

## prediction.shape= (10000,) , y_label_test.shape= (10000, 1)
# 將y_label_test 轉為 一行, 多個 columns
y_label_test.reshape(-1)

import pandas as pd
print(label_dict)
crosstab1 = pd.crosstab(y_label_test.reshape(-1),prediction,
            rownames=['label'],colnames=['predict'])
print()
print("-----crosstab1------")
print(crosstab1)


# -----crosstab1------
# predict    0    1    2    3    4    5    6    7    8    9
# label
# 0        742   13   45   22   29    7   28    9   53   52
# 1         10  814    8   12    7   13   24    3   13   96
# 2         56    3  541   62  121   71  114   23    3    6
# 3         13    7   38  505   82  179  141   16    6   13
# 4          7    2   33   51  736   35  102   26    7    1
# 5          6    1   30  160   63  656   62   15    1    6
# 6          0    2   13   27   13   18  923    1    1    2
# 7          8    0   27   42   93   86   29  709    0    6
# 8         45   40   22   29   16   10   23    2  778   35
# 9         22   62    4   27    6   19   22    8   15  815

# 將模型儲存為 JSON

import os
if not os.path.exists('SaveModel'):
    os.makedirs('SaveModel')

model_json = model.to_json()
with open("SaveModel/cifarCnnModelnew.json", "w") as json_file:
    json_file.write(model_json)

# 將模型儲存為 YAML
model_yaml = model.to_yaml()
with open("SaveModel/cifarCnnModelnew.yaml", "w") as yaml_file:
    yaml_file.write(model_yaml)

# 將模型儲存為 h5
model.save_weights("SaveModel/cifarCnnModelnew.h5")
print("Saved model to disk")
Train on 40000 samples, validate on 10000 samples
Epoch 1/10
40000/40000 [==============================] - 162s 4ms/step - loss: 1.0503 - accuracy: 0.6292 - val_loss: 1.1154 - val_accuracy: 0.6282
Epoch 2/10
40000/40000 [==============================] - 167s 4ms/step - loss: 1.0259 - accuracy: 0.6337 - val_loss: 1.0459 - val_accuracy: 0.6620
Epoch 3/10
40000/40000 [==============================] - 177s 4ms/step - loss: 0.9121 - accuracy: 0.6802 - val_loss: 0.9687 - val_accuracy: 0.6851
Epoch 4/10
40000/40000 [==============================] - 159s 4ms/step - loss: 0.8165 - accuracy: 0.7133 - val_loss: 0.9097 - val_accuracy: 0.7079
Epoch 5/10
40000/40000 [==============================] - 158s 4ms/step - loss: 0.7338 - accuracy: 0.7423 - val_loss: 0.8498 - val_accuracy: 0.7269
Epoch 6/10
40000/40000 [==============================] - 159s 4ms/step - loss: 0.6554 - accuracy: 0.7695 - val_loss: 0.8093 - val_accuracy: 0.7297
Epoch 7/10
40000/40000 [==============================] - 149s 4ms/step - loss: 0.5759 - accuracy: 0.7978 - val_loss: 0.8047 - val_accuracy: 0.7312
Epoch 8/10
40000/40000 [==============================] - 152s 4ms/step - loss: 0.5092 - accuracy: 0.8216 - val_loss: 0.7822 - val_accuracy: 0.7367
Epoch 9/10
40000/40000 [==============================] - 146s 4ms/step - loss: 0.4505 - accuracy: 0.8414 - val_loss: 0.7737 - val_accuracy: 0.7375
Epoch 10/10
40000/40000 [==============================] - 160s 4ms/step - loss: 0.3891 - accuracy: 0.8638 - val_loss: 0.7935 - val_accuracy: 0.7317
scores[1]= 0.7218999862670898

cifar-10 CNN 三次卷積

為增加正確率,修改為三次卷積

epochs 改為 50 次,但這樣會讓程式要跑很久,可先用 1 測試

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 32, 32, 32)        896
_________________________________________________________________
dropout_1 (Dropout)          (None, 32, 32, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 32, 32)        9248
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 16, 16, 32)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 16, 16, 64)        18496
_________________________________________________________________
dropout_2 (Dropout)          (None, 16, 16, 64)        0
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 16, 16, 64)        36928
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 64)          0
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 8, 8, 128)         73856
_________________________________________________________________
dropout_3 (Dropout)          (None, 8, 8, 128)         0
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 8, 8, 128)         147584
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 4, 4, 128)         0
_________________________________________________________________
flatten_1 (Flatten)          (None, 2048)              0
_________________________________________________________________
dropout_4 (Dropout)          (None, 2048)              0
_________________________________________________________________
dense_1 (Dense)              (None, 2500)              5122500
_________________________________________________________________
dropout_5 (Dropout)          (None, 2500)              0
_________________________________________________________________
dense_2 (Dense)              (None, 1500)              3751500
_________________________________________________________________
dropout_6 (Dropout)          (None, 1500)              0
_________________________________________________________________
dense_3 (Dense)              (None, 10)                15010
=================================================================
Total params: 9,176,018
Trainable params: 9,176,018
Non-trainable params: 0
import numpy
from keras.datasets import cifar10
import numpy as np
np.random.seed(10)

###########
# 資料準備,載入 cifar10
# 資料會放在 ~/.keras/datasets/cifar-10-batches-py
(x_img_train,y_label_train), (x_img_test, y_label_test)=cifar10.load_data()

###########
# 對圖片進行預處理

# normalize 標準化,可提升模型的準確度
x_img_train_normalize = x_img_train.astype('float32') / 255.0
x_img_test_normalize = x_img_test.astype('float32') / 255.0

## 將 label 轉為 one hot encoding
from keras.utils import np_utils
y_label_train_OneHot = np_utils.to_categorical(y_label_train)
y_label_test_OneHot = np_utils.to_categorical(y_label_test)

#########
# 建立模型

from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D

model = Sequential()

#卷積層 1與池化層1
model.add(Conv2D(filters=32,kernel_size=(3, 3),input_shape=(32, 32,3),
                 activation='relu', padding='same'))
model.add(Dropout(0.3))
model.add(Conv2D(filters=32, kernel_size=(3, 3),
                 activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))


#卷積層2與池化層2
model.add(Conv2D(filters=64, kernel_size=(3, 3),
                 activation='relu', padding='same'))
model.add(Dropout(0.3))
model.add(Conv2D(filters=64, kernel_size=(3, 3),
                 activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))


#卷積層3與池化層3
model.add(Conv2D(filters=128, kernel_size=(3, 3),
                 activation='relu', padding='same'))
model.add(Dropout(0.3))
model.add(Conv2D(filters=128, kernel_size=(3, 3),
                 activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))


#Step3  建立神經網路(平坦層、隱藏層、輸出層)
model.add(Flatten())
model.add(Dropout(0.3))
model.add(Dense(2500, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(1500, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(10, activation='softmax'))

print(model.summary())

####################
import matplotlib.pyplot as plt
def show_train_history(train_acc,test_acc, filename):
    plt.clf()
    plt.gcf()
    plt.plot(train_history.history[train_acc])
    plt.plot(train_history.history[test_acc])
    plt.title('Train History')
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig(filename)

#### 判斷是否可載入已經訓練好的模型
try:
    model.load_weights("SaveModel/cifarCnnModelnew.h5")
    print("載入模型成功!繼續訓練模型")
except :
    print("載入模型失敗!開始訓練一個新模型")


#### 進行訓練

model.compile(loss='categorical_crossentropy',
              optimizer='adam', metrics=['accuracy'])

train_history=model.fit(x_img_train_normalize, y_label_train_OneHot,
                        validation_split=0.2,
                        epochs=1, batch_size=300, verbose=1)

show_train_history('accuracy','val_accuracy', 'accuracy.png')
show_train_history('loss','val_loss', 'loss.png')

#######
# 評估模型準確率
scores = model.evaluate(x_img_test_normalize,
                        y_label_test_OneHot, verbose=0)
print("scores[1]=", scores[1])

## 進行預測
prediction=model.predict_classes(x_img_test_normalize)

###########
# 查看多筆資料與 label

# 定義 label_dict
label_dict={0:"airplane",1:"automobile",2:"bird",3:"cat",4:"deer",
            5:"dog",6:"frog",7:"horse",8:"ship",9:"truck"}

# 產生圖片, label, prediction 的 preview
import matplotlib.pyplot as plt
def plot_images_labels_prediction(images,labels,prediction,idx,filename,num=10):
    plt.clf()
    fig = plt.gcf()
    fig.set_size_inches(12, 14)
    if num>25: num=25
    for i in range(0, num):
        ax=plt.subplot(5,5, 1+i)
        ax.imshow(images[idx],cmap='binary')

        title=str(i)+','+label_dict[labels[i][0]]
        if len(prediction)>0:
            title+='=>'+label_dict[prediction[i]]

        ax.set_title(title,fontsize=10)
        ax.set_xticks([]);ax.set_yticks([])
        idx+=1
    plt.savefig(filename)

## 列印前 10 筆預測結果
plot_images_labels_prediction(x_img_test,y_label_test,prediction,0,'prediction.png', num=10)


# 查看預測機率
Predicted_Probability=model.predict(x_img_test_normalize)

# y: 真實值
# prediciton: 預測結果
# x_img: 預測的影像
# Predicted_Probability: 預測機率
# i: 資料 index
def show_Predicted_Probability(y,prediction,
                               x_img,Predicted_Probability,i):
    print('-------------------')
    print('label:',label_dict[y[i][0]],
          'predict:',label_dict[prediction[i]])
    plt.clf()
    fig = plt.gcf()
    plt.figure(figsize=(2,2))
    plt.imshow(np.reshape(x_img_test[i],(32, 32,3)))
    plt.savefig(""+str(i)+".png")
    for j in range(10):
        print(label_dict[j]+
              ' Probability:%1.9f'%(Predicted_Probability[i][j]))

show_Predicted_Probability(y_label_test,prediction,
                           x_img_test,Predicted_Probability,0)

show_Predicted_Probability(y_label_test,prediction,
                           x_img_test,Predicted_Probability,3)

#####
# confusion matrix

print("prediction.shape=", str(prediction.shape), ", y_label_test.shape=",str(y_label_test.shape))
# 將y_label_test 轉為 一行, 多個 columns
y_label_test.reshape(-1)

import pandas as pd
print(label_dict)
crosstab1 = pd.crosstab(y_label_test.reshape(-1),prediction,
            rownames=['label'],colnames=['predict'])
print()
print("-----crosstab1------")
print(crosstab1)

# 將模型儲存為 JSON

import os
if not os.path.exists('SaveModel'):
    os.makedirs('SaveModel')

model_json = model.to_json()
with open("SaveModel/cifarCnnModelnew.json", "w") as json_file:
    json_file.write(model_json)

# 將模型儲存為 YAML
model_yaml = model.to_yaml()
with open("SaveModel/cifarCnnModelnew.yaml", "w") as yaml_file:
    yaml_file.write(model_yaml)

# 將模型儲存為 h5
model.save_weights("SaveModel/cifarCnnModelnew.h5")
print("Saved model to disk")

Note

程式放到 CUDA 機器上,安裝 tensorflow-gpu 出現 error 的解決方式

  1. 在 tensorflow-gpu 出現 error

    Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

    ref: https://davistseng.blogspot.com/2019/11/tensorflow-2.html

import tensorflow as tf
def solve_cudnn_error():
    gpus = tf.config.experimental.list_physical_devices('GPU')
    if gpus:
        try:
            # Currently, memory growth needs to be the same across GPUs
            for gpu in gpus:
                tf.config.experimental.set_memory_growth(gpu, True)
            logical_gpus = tf.config.experimental.list_logical_devices('GPU')
            print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
        except RuntimeError as e:
            # Memory growth must be set before GPUs have been initialized
            print(e)

solve_cudnn_error()
  1. pandas error: No module named '_bz2'

    ref: https://stackoverflow.com/questions/12806122/missing-python-bz2-module

    cp /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so  /usr/local/lib/python3.6/lib-dynload/

references

TensorFlow+Keras深度學習人工智慧實務應用

2020/10/5

keras Titanic MLP 分析

Titanic 在 1912/4/12 撞上冰山沈沒,乘客與船員共 2224 人,其中 1502 人死亡。接下來以 MLP 預測每一個乘客的存活率。

乘客資料

下載 Titanic 客戶資料

# get titanic data
import urllib.request
import os

import os
if not os.path.exists('data'):
    os.makedirs('data')

url="http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.xls"
filepath="data/titanic3.xls"
if not os.path.isfile(filepath):
    result=urllib.request.urlretrieve(url,filepath)
    print('downloaded:',result)

訓練資料共 1309 筆,資料處理後,會有 9 個特徵欄位,label 欄位是 1: 存活 2: 死亡

透過 pandas 讀取資料並進行預處理

原始資料欄位

欄位 說明 資料說明
survival 是否存活 0: 否, 1: 是
pclass 艙等 1: 頭等艙, 2: 二等艙, 3: 三等艙
name 姓名
sex 性別 female: 女性, male: 男性
age 年齡
sibsp 手足或配偶也在船上的數量
parch 雙親或子女也在船上的數量
ticket 車票號碼
fare 旅客費用
cabin 艙位號碼
embarked 登船港口 C: Cherbourg, Q: Queenstown, S: Southampton

MLP

訓練過程

Train on 943 samples, validate on 105 samples
Epoch 1/30
 - 1s - loss: 0.6894 - acc: 0.5885 - val_loss: 0.6668 - val_acc: 0.7810
Epoch 2/30
 - 0s - loss: 0.6613 - acc: 0.6066 - val_loss: 0.5626 - val_acc: 0.7810
Epoch 3/30
 - 0s - loss: 0.6067 - acc: 0.6585 - val_loss: 0.4871 - val_acc: 0.8190
Epoch 4/30
 - 0s - loss: 0.5551 - acc: 0.7508 - val_loss: 0.4578 - val_acc: 0.7714
Epoch 5/30
 - 0s - loss: 0.5250 - acc: 0.7625 - val_loss: 0.4412 - val_acc: 0.8190
Epoch 6/30
 - 0s - loss: 0.5076 - acc: 0.7550 - val_loss: 0.4274 - val_acc: 0.8190
Epoch 7/30
 - 0s - loss: 0.5013 - acc: 0.7688 - val_loss: 0.4276 - val_acc: 0.8190
Epoch 8/30
 - 0s - loss: 0.4936 - acc: 0.7688 - val_loss: 0.4255 - val_acc: 0.8190
Epoch 9/30
 - 0s - loss: 0.4897 - acc: 0.7699 - val_loss: 0.4211 - val_acc: 0.8190
Epoch 10/30
 - 0s - loss: 0.4851 - acc: 0.7731 - val_loss: 0.4228 - val_acc: 0.8190
Epoch 11/30
 - 0s - loss: 0.4819 - acc: 0.7699 - val_loss: 0.4161 - val_acc: 0.8190
Epoch 12/30
 - 0s - loss: 0.4796 - acc: 0.7709 - val_loss: 0.4142 - val_acc: 0.8381
Epoch 13/30
 - 0s - loss: 0.4773 - acc: 0.7762 - val_loss: 0.4168 - val_acc: 0.8190
Epoch 14/30
 - 0s - loss: 0.4733 - acc: 0.7805 - val_loss: 0.4130 - val_acc: 0.8190
Epoch 15/30
 - 0s - loss: 0.4732 - acc: 0.7752 - val_loss: 0.4120 - val_acc: 0.8286
Epoch 16/30
 - 0s - loss: 0.4709 - acc: 0.7815 - val_loss: 0.4107 - val_acc: 0.8286
Epoch 17/30
 - 0s - loss: 0.4692 - acc: 0.7815 - val_loss: 0.4125 - val_acc: 0.8476
Epoch 18/30
 - 0s - loss: 0.4677 - acc: 0.7847 - val_loss: 0.4134 - val_acc: 0.8381
Epoch 19/30
 - 0s - loss: 0.4670 - acc: 0.7826 - val_loss: 0.4092 - val_acc: 0.8571
Epoch 20/30
 - 0s - loss: 0.4645 - acc: 0.7741 - val_loss: 0.4109 - val_acc: 0.8571
Epoch 21/30
 - 0s - loss: 0.4646 - acc: 0.7858 - val_loss: 0.4123 - val_acc: 0.8571
Epoch 22/30
 - 0s - loss: 0.4615 - acc: 0.7953 - val_loss: 0.4178 - val_acc: 0.8095
Epoch 23/30
 - 0s - loss: 0.4614 - acc: 0.7858 - val_loss: 0.4111 - val_acc: 0.8571
Epoch 24/30
 - 0s - loss: 0.4611 - acc: 0.7900 - val_loss: 0.4128 - val_acc: 0.8571
Epoch 25/30
 - 0s - loss: 0.4610 - acc: 0.7847 - val_loss: 0.4170 - val_acc: 0.8381
Epoch 26/30
 - 0s - loss: 0.4574 - acc: 0.7911 - val_loss: 0.4122 - val_acc: 0.8571
Epoch 27/30
 - 0s - loss: 0.4580 - acc: 0.7953 - val_loss: 0.4161 - val_acc: 0.8286
Epoch 28/30
 - 0s - loss: 0.4561 - acc: 0.7985 - val_loss: 0.4168 - val_acc: 0.8381
Epoch 29/30
 - 0s - loss: 0.4582 - acc: 0.7879 - val_loss: 0.4154 - val_acc: 0.8381
Epoch 30/30
 - 0s - loss: 0.4574 - acc: 0.7794 - val_loss: 0.4181 - val_acc: 0.8286
261/261 [==============================] - 0s 20us/step

References

TensorFlow+Keras深度學習人工智慧實務應用