特徴抽出のワークフロー

この例では次を使用します。

この例では、イメージデータからの特徴抽出を行う完全なワークフローを示します。

データの取得

この例では、手書き数字のイメージから構成されている MNIST イメージデータ [1] を使用します。イメージはグレースケールの 28 x 28 ピクセルです。各イメージには、イメージが表す数字である 0 ～ 9 のラベルが関連付けられています。

はじめに、イメージとラベルのデータを以下から取得します。

http://yann.lecun.com/exdb/mnist/

ファイルを解凍します。この長い例でパフォーマンスを向上させるため、検定データを学習データとして、学習データを検定データとして使用します。

imageFileName = 't10k-images.idx3-ubyte';
labelFileName = 't10k-labels.idx1-ubyte';

ファイルを処理してワークスペースに読み込みます。この処理関数のコードは、この例の終わりで示します。

[Xtrain,LabelTrain] = processMNISTdata(imageFileName,labelFileName);

Read MNIST image data...
Number of images in the dataset:  10000 ...
Each image is of 28 by 28 pixels...
The image data is read to a matrix of dimensions:  10000 by  784...
End of reading image data.

Read MNIST label data...
Number of labels in the dataset:  10000 ...
The label data is read to a matrix of dimensions:  10000 by  1...
End of reading label data.

いくつかのイメージを表示します。

rng('default') % For reproducibility
numrows = size(Xtrain,1);
ims = randi(numrows,4,1);
imgs = Xtrain(ims,:);
for i = 1:4
    pp{i} = reshape(imgs(i,:),28,28);
end
ppf = [pp{1},pp{2};pp{3},pp{4}];
imshow(ppf);

新しい特徴次元の選択

抽出する特徴量の個数の選択には、いくつかの考慮事項があります。

特徴量が多くなると、メモリと計算時間が増える。
特徴量が少ないと、分類器が不適切になる可能性がある。

この例では、100 個の特徴量を選択します。

q = 100;

特徴量の抽出

特徴抽出関数には、sparsefilt と rica の 2 つがあります。関数 sparsefilt から始めます。抽出の時間がかかりすぎないようにするため、反復回数を 10 に設定します。

通常は、数回から数百回の反復で sparsefilt アルゴリズムを実行すると良好な結果が得られます。反復回数が多すぎると、一種の過適合問題が発生し、分類精度が低下する可能性があります。

10 回の反復で sparsefilt を使用して、スパースフィルターモデルを取得します。

Mdl = sparsefilt(Xtrain,q,'IterationLimit',10);

Warning: Solver LBFGS was not able to converge to a solution.

sparsefilt は、内部の LBFGS オプティマイザーが収束しなかったことを警告しています。オプティマイザーが収束しなかったのは、反復制限を 10 に設定したからです。それでも、この結果を使用して分類器に学習させることができます。

分類器の作成

元のデータを新しい特徴量の表現に変換します。

NewX = transform(Mdl,Xtrain);

変換したデータと LabelTrain の正しい分類ラベルに基づいて線形分類器に学習をさせます。学習済みモデルの精度は、fitcecoc の正則化パラメーター Lambda の影響を受けます。名前と値のペア OptimizeHyperparameters を使用して、Lambda の最適な値を求めてください。この最適化には時間がかかることに注意してください。Parallel Computing Toolbox™ のライセンスがある場合、並列計算を使用して実行を高速化することができます。並列のライセンスがない場合は、このスクリプトを実行する前に UseParallel の呼び出しを削除してください。

t = templateLinear('Solver','lbfgs');
options = struct('UseParallel',true);
Cmdl = fitcecoc(NewX,LabelTrain,'Learners',t, ...
    'OptimizeHyperparameters',{'Lambda'}, ...
    'HyperparameterOptimizationOptions',options);

Copying objective function to workers...
Done copying objective function to workers.
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|    1 |       6 | Best   |      0.5777 |      8.5334 |      0.5777 |      0.5777 |      0.20606 |
|    2 |       5 | Accept |      0.8865 |      8.9062 |      0.2041 |     0.27206 |       8.8234 |
|    3 |       5 | Best   |      0.2041 |      9.7024 |      0.2041 |     0.27206 |     0.026804 |
|    4 |       6 | Best   |      0.1077 |      14.629 |      0.1077 |     0.10773 |   1.7309e-09 |
|    5 |       6 | Best   |      0.0962 |      15.767 |      0.0962 |    0.096203 |    0.0002442 |
|    6 |       6 | Accept |      0.1999 |      6.4363 |      0.0962 |     0.09622 |     0.024862 |
|    7 |       6 | Accept |      0.2074 |      6.4171 |      0.0962 |    0.096222 |     0.029034 |
|    8 |       6 | Accept |      0.1065 |      12.974 |      0.0962 |    0.096222 |    2.037e-08 |
|    9 |       6 | Accept |      0.0977 |      22.976 |      0.0962 |    0.096216 |   8.0495e-06 |
|   10 |       6 | Accept |      0.1237 |      8.5033 |      0.0962 |    0.096199 |    0.0029745 |
|   11 |       6 | Accept |      0.1076 |      10.653 |      0.0962 |    0.096208 |   0.00080903 |
|   12 |       6 | Accept |      0.1034 |      16.761 |      0.0962 |      0.0962 |   3.2145e-07 |
|   13 |       6 | Best   |      0.0933 |      16.715 |      0.0933 |    0.093293 |   6.3327e-05 |
|   14 |       6 | Accept |       0.109 |      12.946 |      0.0933 |     0.09328 |   5.7887e-09 |
|   15 |       6 | Accept |      0.0994 |      18.805 |      0.0933 |    0.093312 |   1.8981e-06 |
|   16 |       6 | Accept |       0.106 |      15.088 |      0.0933 |    0.093306 |   7.4684e-08 |
|   17 |       6 | Accept |      0.0952 |      20.372 |      0.0933 |    0.093285 |   2.2831e-05 |
|   18 |       6 | Accept |      0.0933 |      14.528 |      0.0933 |    0.093459 |   0.00013097 |
|   19 |       6 | Accept |      0.1082 |      12.764 |      0.0933 |    0.093458 |   1.0001e-09 |
|   20 |       6 | Best   |      0.0915 |      16.157 |      0.0915 |    0.092391 |   8.3234e-05 |
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|   21 |       6 | Accept |      0.8865 |      6.6373 |      0.0915 |    0.092387 |       1.6749 |
|   22 |       6 | Accept |      0.0929 |      17.306 |      0.0915 |    0.092457 |   0.00010668 |
|   23 |       6 | Accept |      0.0937 |      19.046 |      0.0915 |    0.092535 |   5.0962e-05 |
|   24 |       6 | Accept |      0.0916 |      17.932 |      0.0915 |    0.092306 |    9.023e-05 |
|   25 |       6 | Accept |      0.0935 |       17.53 |      0.0915 |    0.092431 |   0.00011726 |
|   26 |       6 | Accept |      0.1474 |      8.3795 |      0.0915 |    0.092397 |     0.006997 |
|   27 |       6 | Accept |      0.0939 |      19.188 |      0.0915 |    0.092427 |   5.2557e-05 |
|   28 |       6 | Accept |      0.1147 |      10.686 |      0.0915 |    0.092432 |    0.0015036 |
|   29 |       6 | Accept |      0.1049 |      16.609 |      0.0915 |    0.092434 |   1.4871e-07 |
|   30 |       6 | Accept |      0.1069 |      13.929 |      0.0915 |    0.092435 |   1.0899e-08 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 83.1976 seconds.
Total objective function evaluation time: 416.8767

Best observed feasible point:
      Lambda  
    __________

    8.3234e-05

Observed objective function value = 0.0915
Estimated objective function value = 0.09245
Function evaluation time = 16.1569

Best estimated feasible point (according to models):
     Lambda  
    _________

    9.023e-05

Estimated objective function value = 0.092435
Estimated function evaluation time = 17.0972

分類器の評価

検定データに適用したときの分類器の誤差をチェックします。はじめに、検定データを読み込みます。

imageFileName = 'train-images.idx3-ubyte';
labelFileName = 'train-labels.idx1-ubyte';
[Xtest,LabelTest] = processMNISTdata(imageFileName,labelFileName);

Read MNIST image data...
Number of images in the dataset:  60000 ...
Each image is of 28 by 28 pixels...
The image data is read to a matrix of dimensions:  60000 by  784...
End of reading image data.

Read MNIST label data...
Number of labels in the dataset:  60000 ...
The label data is read to a matrix of dimensions:  60000 by  1...
End of reading label data.

分類器を検定データに適用したときの分類損失を計算します。

TestX = transform(Mdl,Xtest);
Loss = loss(Cmdl,TestX,LabelTest)

Loss =

    0.1009

この変換により、元のデータで学習をさせたものより優れた分類器が生成されたでしょうか。元の学習データに基づいて分類器を作成し、損失を評価します。

Omdl = fitcecoc(Xtrain,LabelTrain,'Learners',t, ...
    'OptimizeHyperparameters',{'Lambda'}, ...
    'HyperparameterOptimizationOptions',options);
Losso = loss(Omdl,Xtest,LabelTest)

Copying objective function to workers...
Done copying objective function to workers.
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|    1 |       5 | Best   |      0.0779 |      46.965 |      0.0779 |      0.0779 |   5.7933e-08 |
|    2 |       5 | Accept |      0.0779 |      47.003 |      0.0779 |      0.0779 |   3.8643e-09 |
|    3 |       5 | Accept |      0.0779 |      47.068 |      0.0779 |      0.0779 |   1.3269e-06 |
|    4 |       6 | Accept |       0.078 |      60.714 |      0.0779 |    0.077925 |   3.0332e-05 |
|    5 |       6 | Accept |      0.0787 |      133.21 |      0.0779 |      0.0779 |     0.011605 |
|    6 |       6 | Best   |      0.0775 |      135.97 |      0.0775 |    0.077983 |   0.00020291 |
|    7 |       6 | Accept |      0.0779 |      44.642 |      0.0775 |    0.077971 |    5.735e-08 |
|    8 |       6 | Accept |      0.0785 |      123.19 |      0.0775 |      0.0775 |     0.024589 |
|    9 |       6 | Accept |      0.0779 |      43.574 |      0.0775 |      0.0775 |   1.0042e-09 |
|   10 |       6 | Accept |      0.0779 |      43.038 |      0.0775 |      0.0775 |   4.7227e-06 |
|   11 |       6 | Best   |      0.0774 |      137.51 |      0.0774 |    0.077451 |   0.00021639 |
|   12 |       6 | Accept |      0.0779 |       44.07 |      0.0774 |    0.077452 |   6.7132e-09 |
|   13 |       6 | Accept |      0.0779 |      44.822 |      0.0774 |    0.077453 |    2.873e-07 |
|   14 |       6 | Best   |      0.0744 |      233.12 |      0.0744 |    0.074402 |        6.805 |
|   15 |       6 | Accept |      0.0778 |      140.49 |      0.0744 |    0.074406 |      0.66889 |
|   16 |       6 | Accept |      0.0774 |      149.32 |      0.0744 |    0.074405 |    0.0002769 |
|   17 |       6 | Accept |      0.0774 |         155 |      0.0744 |    0.074404 |   0.00046083 |
|   18 |       6 | Accept |      0.0765 |      152.63 |      0.0744 |    0.074687 |   0.00027101 |
|   19 |       6 | Accept |      0.0768 |      156.32 |      0.0744 |    0.077558 |   0.00026573 |
|   20 |       6 | Best   |      0.0725 |      255.51 |      0.0725 |    0.073249 |       9.9961 |
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|   21 |       6 | Best   |      0.0723 |       221.5 |      0.0723 |    0.073161 |        4.212 |
|   22 |       6 | Accept |      0.0732 |      259.51 |      0.0723 |    0.073166 |       9.9916 |
|   23 |       6 | Best   |       0.072 |      261.94 |       0.072 |    0.072848 |       9.9883 |
|   24 |       6 | Accept |      0.0778 |      122.56 |       0.072 |    0.072854 |      0.13413 |
|   25 |       6 | Accept |      0.0733 |      258.54 |       0.072 |    0.072946 |       9.9904 |
|   26 |       6 | Accept |      0.0746 |      244.53 |       0.072 |    0.073144 |       7.0911 |
|   27 |       6 | Accept |      0.0779 |      44.573 |       0.072 |    0.073134 |   2.1183e-08 |
|   28 |       6 | Accept |       0.078 |      45.478 |       0.072 |    0.073126 |   1.1663e-05 |
|   29 |       6 | Accept |      0.0779 |      43.954 |       0.072 |    0.073118 |    1.336e-07 |
|   30 |       6 | Accept |      0.0779 |      44.574 |       0.072 |    0.073112 |   1.7282e-09 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 690.8688 seconds.
Total objective function evaluation time: 3741.3176

Best observed feasible point:
    Lambda
    ______

    9.9883

Observed objective function value = 0.072
Estimated objective function value = 0.073112
Function evaluation time = 261.9357

Best estimated feasible point (according to models):
    Lambda
    ______

    9.9961

Estimated objective function value = 0.073112
Estimated function evaluation time = 257.9556


Losso =

    0.0865

スパースフィルターに基づく分類器では、元のデータに基づく分類器より損失がわずかに大きくなっています。しかし、元のデータでは 784 個の特徴量を使用していましたが、この分類器は 100 個の特徴量しか使用しておらず、はるかに短時間で作成できます。より優れたスパースフィルター分類器を作成するため、q を 100 から 200 に増やします。これでも、784 よりはるかに少ない値です。

q = 200;
Mdl2 = sparsefilt(Xtrain,q,'IterationLimit',10);
NewX = transform(Mdl2,Xtrain);
TestX = transform(Mdl2,Xtest);
Cmdl = fitcecoc(NewX,LabelTrain,'Learners',t, ...
    'OptimizeHyperparameters',{'Lambda'}, ...
    'HyperparameterOptimizationOptions',options);
Loss2 = loss(Cmdl,TestX,LabelTest)

Warning: Solver LBFGS was not able to converge to a solution. 
Copying objective function to workers...
Done copying objective function to workers.
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|    1 |       5 | Best   |      0.8865 |      7.3578 |      0.8865 |      0.8865 |         1.93 |
|    2 |       5 | Accept |      0.8865 |      7.3408 |      0.8865 |      0.8865 |       2.5549 |
|    3 |       6 | Best   |      0.0693 |      9.0077 |      0.0693 |    0.069376 |   9.9515e-09 |
|    4 |       5 | Accept |      0.0705 |      9.1067 |      0.0693 |    0.069374 |   1.2123e-08 |
|    5 |       5 | Accept |      0.1489 |      9.5685 |      0.0693 |    0.069374 |     0.015542 |
|    6 |       6 | Accept |      0.8865 |      7.5032 |      0.0693 |     0.06943 |       4.7067 |
|    7 |       6 | Accept |       0.071 |      8.8044 |      0.0693 |    0.069591 |   5.0861e-09 |
|    8 |       6 | Accept |      0.0715 |      8.9517 |      0.0693 |    0.070048 |    1.001e-09 |
|    9 |       6 | Accept |      0.0833 |      14.393 |      0.0693 |    0.069861 |    0.0014191 |
|   10 |       6 | Best   |      0.0594 |      25.565 |      0.0594 |    0.059458 |    6.767e-05 |
|   11 |       6 | Accept |      0.0651 |      20.074 |      0.0594 |    0.059463 |    8.078e-07 |
|   12 |       6 | Accept |      0.0695 |      14.495 |      0.0594 |    0.059473 |   1.0381e-07 |
|   13 |       6 | Accept |      0.1042 |      12.085 |      0.0594 |    0.059386 |    0.0039745 |
|   14 |       6 | Accept |       0.065 |      20.235 |      0.0594 |    0.059416 |   0.00031759 |
|   15 |       6 | Accept |      0.0705 |      10.929 |      0.0594 |    0.059416 |   3.6503e-08 |
|   16 |       6 | Accept |      0.0637 |      30.593 |      0.0594 |    0.059449 |   8.8718e-06 |
|   17 |       6 | Accept |       0.064 |      25.084 |      0.0594 |    0.059464 |   2.6286e-06 |
|   18 |       6 | Accept |      0.0605 |      31.964 |      0.0594 |    0.059387 |    2.459e-05 |
|   19 |       6 | Accept |      0.0606 |      23.149 |      0.0594 |    0.059312 |    0.0001464 |
|   20 |       6 | Accept |      0.0602 |      32.178 |      0.0594 |    0.059874 |   4.1437e-05 |
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|   21 |       6 | Accept |      0.0594 |      27.686 |      0.0594 |    0.059453 |   8.0717e-05 |
|   22 |       6 | Accept |      0.0612 |      33.427 |      0.0594 |    0.059476 |   1.6878e-05 |
|   23 |       6 | Accept |      0.0673 |      17.444 |      0.0594 |    0.059475 |   3.1788e-07 |
|   24 |       6 | Best   |      0.0593 |      26.262 |      0.0593 |     0.05944 |   7.8179e-05 |
|   25 |       6 | Accept |       0.248 |      7.6345 |      0.0593 |    0.059409 |     0.095654 |
|   26 |       6 | Accept |      0.0598 |      28.536 |      0.0593 |    0.059465 |   5.0819e-05 |
|   27 |       6 | Accept |      0.0701 |      9.0545 |      0.0593 |    0.059466 |   1.8937e-09 |
|   28 |       5 | Accept |      0.7081 |      7.1176 |      0.0593 |    0.059372 |      0.30394 |
|   29 |       5 | Accept |      0.0676 |      11.782 |      0.0593 |    0.059372 |   6.1136e-08 |
|   30 |       3 | Accept |        0.06 |      23.556 |      0.0593 |    0.059422 |   0.00010144 |
|   31 |       3 | Accept |      0.0725 |      16.069 |      0.0593 |    0.059422 |   0.00069403 |
|   32 |       3 | Accept |      0.1928 |      8.3732 |      0.0593 |    0.059422 |     0.040402 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 32
Total elapsed time: 97.7946 seconds.
Total objective function evaluation time: 545.3255

Best observed feasible point:
      Lambda  
    __________

    7.8179e-05

Observed objective function value = 0.0593
Estimated objective function value = 0.059422
Function evaluation time = 26.2624

Best estimated feasible point (according to models):
      Lambda  
    __________

    7.8179e-05

Estimated objective function value = 0.059422
Estimated function evaluation time = 26.508


Loss2 =

    0.0682

今回は、元のデータの分類器より分類損失が小さくなっています。

RICA を試す

他の特徴抽出関数 rica を試します。200 個の特徴量を抽出し、分類器を作成して、検定データに対する損失を確認します。rica は sparsefilt より反復回数を増やす方が性能が向上する可能性があるので、関数 rica では反復回数を増やします。

多くの場合、特徴抽出を行う前に、データの前処理ステップとして入力データを "事前に白色化" します。前置白色化ステップには、相関除去および標準化という 2 つの変換が含まれます。これにより、予測子がゼロ平均と単位共分散をもつようになります。rica は標準化変換のみをサポートします。予測子がゼロ平均と単位共分散をもつようにするには、名前と値のペアの引数 Standardize を使用します。あるいは、sparsefilt または rica を呼び出す前に zscore 変換を適用して、コントラスト正規化用にイメージを個別に変換することもできます。

Mdl3 = rica(Xtrain,q,'IterationLimit',400,'Standardize',true);
NewX = transform(Mdl3,Xtrain);
TestX = transform(Mdl3,Xtest);
Cmdl = fitcecoc(NewX,LabelTrain,'Learners',t, ...
    'OptimizeHyperparameters',{'Lambda'}, ...
    'HyperparameterOptimizationOptions',options);
Loss3 = loss(Cmdl,TestX,LabelTest)

Warning: Solver LBFGS was not able to converge to a solution. 
Copying objective function to workers...
Done copying objective function to workers.
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|    1 |       6 | Best   |      0.1179 |      12.012 |      0.1179 |      0.1179 |       8.4727 |
|    2 |       6 | Best   |       0.082 |      13.384 |       0.082 |    0.083897 |   4.3291e-09 |
|    3 |       6 | Best   |      0.0809 |      18.917 |      0.0809 |    0.080902 |    1.738e-05 |
|    4 |       6 | Accept |      0.0821 |      19.172 |      0.0809 |     0.08091 |   3.8101e-06 |
|    5 |       6 | Accept |      0.0921 |      14.445 |      0.0809 |    0.086349 |       2.3753 |
|    6 |       6 | Accept |      0.0809 |      13.393 |      0.0809 |    0.083836 |   1.3757e-08 |
|    7 |       6 | Best   |       0.076 |      28.075 |       0.076 |    0.081808 |   0.00027773 |
|    8 |       6 | Best   |      0.0758 |      29.686 |      0.0758 |    0.078829 |   0.00068195 |
|    9 |       6 | Accept |      0.0829 |      13.373 |      0.0758 |    0.078733 |   1.7543e-07 |
|   10 |       6 | Accept |      0.0826 |      14.031 |      0.0758 |    0.078512 |   1.0045e-09 |
|   11 |       6 | Accept |      0.0817 |      13.662 |      0.0758 |    0.078077 |   2.4568e-08 |
|   12 |       6 | Accept |      0.0799 |      19.311 |      0.0758 |    0.077658 |   1.4061e-05 |
|   13 |       6 | Best   |       0.065 |      25.148 |       0.065 |    0.064974 |     0.060326 |
|   14 |       6 | Accept |      0.0787 |      23.434 |       0.065 |    0.064947 |   0.00012407 |
|   15 |       6 | Accept |       0.072 |      19.167 |       0.065 |    0.064997 |      0.43899 |
|   16 |       6 | Accept |       0.073 |       28.39 |       0.065 |    0.065053 |    0.0023721 |
|   17 |       6 | Accept |      0.0787 |      29.887 |       0.065 |    0.064928 |   0.00042914 |
|   18 |       6 | Accept |      0.0662 |      26.374 |       0.065 |    0.064295 |    0.0077638 |
|   19 |       6 | Accept |      0.0652 |      24.937 |       0.065 |    0.064502 |     0.087389 |
|   20 |       6 | Accept |      0.0655 |      25.416 |       0.065 |    0.064762 |     0.072931 |
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|   21 |       6 | Best   |      0.0645 |      25.529 |      0.0645 |    0.064691 |     0.059245 |
|   22 |       6 | Accept |       0.065 |      23.832 |      0.0645 |     0.06474 |     0.025521 |
|   23 |       6 | Accept |      0.0819 |      20.343 |      0.0645 |    0.064732 |   7.2593e-07 |
|   24 |       6 | Accept |      0.0664 |      23.732 |      0.0645 |    0.064718 |       0.1534 |
|   25 |       6 | Accept |      0.0651 |      24.796 |      0.0645 |    0.064693 |     0.038371 |
|   26 |       6 | Accept |      0.0651 |      25.449 |      0.0645 |    0.064613 |     0.014318 |
|   27 |       6 | Accept |      0.0652 |      25.092 |      0.0645 |    0.064713 |     0.037107 |
|   28 |       6 | Accept |      0.0645 |      24.404 |      0.0645 |      0.0647 |     0.042959 |
|   29 |       6 | Accept |      0.0649 |      24.704 |      0.0645 |    0.064729 |     0.042776 |
|   30 |       6 | Accept |      0.0652 |      24.341 |      0.0645 |    0.064786 |     0.035788 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 124.9755 seconds.
Total objective function evaluation time: 654.4364

Best observed feasible point:
     Lambda 
    ________

    0.059245

Observed objective function value = 0.0645
Estimated objective function value = 0.064932
Function evaluation time = 25.5294

Best estimated feasible point (according to models):
     Lambda 
    ________

    0.042776

Estimated objective function value = 0.064786
Estimated function evaluation time = 24.7849


Loss3 =

    0.0749

rica に基づく分類器では、スパースフィルター分類器よりわずかに検定損失が大きくなっています。

より多くの特徴量を試す

特徴抽出関数にはいくつかの調整パラメーターがあります。結果に影響を与える可能性があるパラメーターの 1 つは、要求する特徴量の個数です。前に試した 200 個の特徴量や元のデータの 784 個の特徴量ではなく 1000 個の特徴量に基づいた場合に、分類器がどの程度良好に機能するかを調べます。元のデータより多くの特徴量を使用することを "過完備" 学習と呼びます。逆に、少ない特徴量を使用することを "劣完備" 学習と呼びます。過完備学習では分類器の精度が向上する可能性があり、劣完備学習ではメモリと時間が節約される可能性があります。

q = 1000;
Mdl4 = sparsefilt(Xtrain,q,'IterationLimit',10);
NewX = transform(Mdl4,Xtrain);
TestX = transform(Mdl4,Xtest);
Cmdl = fitcecoc(NewX,LabelTrain,'Learners',t, ...
    'OptimizeHyperparameters',{'Lambda'}, ...
    'HyperparameterOptimizationOptions',options);
Loss4 = loss(Cmdl,TestX,LabelTest)

Warning: Solver LBFGS was not able to converge to a solution. 
Copying objective function to workers...
Done copying objective function to workers.
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|    1 |       6 | Best   |      0.5293 |      39.885 |      0.5293 |      0.5293 |      0.20333 |
|    2 |       6 | Accept |      0.8022 |      43.475 |      0.5293 |     0.66575 |      0.77337 |
|    3 |       6 | Best   |      0.0406 |      52.594 |      0.0406 |     0.11113 |   9.1082e-09 |
|    4 |       6 | Best   |      0.0403 |       54.73 |      0.0403 |    0.060037 |   2.3947e-09 |
|    5 |       6 | Accept |      0.0695 |      124.96 |      0.0403 |    0.040319 |     0.001361 |
|    6 |       6 | Accept |      0.0406 |      53.691 |      0.0403 |    0.040207 |   1.0005e-09 |
|    7 |       6 | Best   |      0.0388 |      178.69 |      0.0388 |    0.038811 |   1.4358e-06 |
|    8 |       6 | Accept |      0.0615 |      138.53 |      0.0388 |    0.038817 |   0.00088731 |
|    9 |       6 | Best   |      0.0385 |       61.81 |      0.0385 |    0.038557 |   7.4709e-08 |
|   10 |       6 | Accept |      0.0399 |      54.198 |      0.0385 |    0.038555 |   2.1909e-08 |
|   11 |       6 | Accept |      0.0402 |      234.55 |      0.0385 |    0.038639 |     0.000101 |
|   12 |       6 | Accept |      0.0431 |      198.09 |      0.0385 |    0.038636 |   0.00018896 |
|   13 |       6 | Accept |      0.0393 |      75.811 |      0.0385 |    0.039016 |   1.1597e-07 |
|   14 |       6 | Accept |      0.0387 |      61.281 |      0.0385 |    0.038908 |   7.0518e-08 |
|   15 |       6 | Accept |      0.0393 |      125.73 |      0.0385 |    0.038931 |   2.8429e-07 |
|   16 |       6 | Accept |      0.0397 |      89.804 |      0.0385 |    0.039106 |   1.4603e-07 |
|   17 |       6 | Accept |      0.0391 |      126.88 |      0.0385 |    0.039081 |   3.0065e-07 |
|   18 |       6 | Accept |      0.0398 |      56.157 |      0.0385 |    0.039123 |   4.1563e-08 |
|   19 |       6 | Accept |      0.0406 |       55.25 |      0.0385 |    0.039122 |   1.0014e-09 |
|   20 |       6 | Accept |      0.0385 |      272.92 |      0.0385 |    0.039127 |    9.568e-06 |
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|   21 |       6 | Accept |      0.0412 |      55.191 |      0.0385 |    0.039124 |   3.3737e-09 |
|   22 |       6 | Accept |      0.0394 |      229.72 |      0.0385 |    0.039117 |   3.2757e-06 |
|   23 |       6 | Best   |      0.0379 |      295.55 |      0.0379 |    0.039116 |   2.8439e-05 |
|   24 |       6 | Accept |      0.0394 |      168.74 |      0.0379 |    0.039111 |    9.778e-07 |
|   25 |       6 | Accept |       0.039 |      281.91 |      0.0379 |    0.039112 |   8.0694e-06 |
|   26 |       6 | Accept |      0.8865 |      54.865 |      0.0379 |    0.038932 |       9.9885 |
|   27 |       6 | Accept |      0.0381 |       300.7 |      0.0379 |    0.037996 |   2.6027e-05 |
|   28 |       6 | Accept |      0.0406 |      54.611 |      0.0379 |    0.037996 |   1.6057e-09 |
|   29 |       6 | Accept |      0.1272 |      76.648 |      0.0379 |    0.037997 |     0.012507 |
|   30 |       6 | Accept |      0.0403 |      57.931 |      0.0379 |    0.037997 |   4.9907e-08 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 724.6036 seconds.
Total objective function evaluation time: 3674.8899

Best observed feasible point:
      Lambda  
    __________

    2.8439e-05

Observed objective function value = 0.0379
Estimated objective function value = 0.03801
Function evaluation time = 295.5515

Best estimated feasible point (according to models):
      Lambda  
    __________

    2.6027e-05

Estimated objective function value = 0.037997
Estimated function evaluation time = 297.6756


Loss4 =

    0.0440

1000 個の特徴量を抽出した過完備スパースフィルターに基づく分類器は、今までにテストした分類器の中で検定損失が最小になります。

Mdl5 = rica(Xtrain,q,'IterationLimit',400,'Standardize',true);
NewX = transform(Mdl5,Xtrain);
TestX = transform(Mdl5,Xtest);
Cmdl = fitcecoc(NewX,LabelTrain,'Learners',t, ...
    'OptimizeHyperparameters',{'Lambda'}, ...
    'HyperparameterOptimizationOptions',options);
Loss5 = loss(Cmdl,TestX,LabelTest)

Warning: Solver LBFGS was not able to converge to a solution. 
Copying objective function to workers...
Done copying objective function to workers.
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|    1 |       6 | Best   |      0.0764 |      46.206 |      0.0764 |      0.0764 |   8.4258e-09 |
|    2 |       6 | Accept |       0.077 |      141.95 |      0.0764 |      0.0767 |   6.9536e-06 |
|    3 |       6 | Accept |      0.0771 |      146.87 |      0.0764 |    0.076414 |   7.3378e-06 |
|    4 |       6 | Best   |      0.0709 |      182.51 |      0.0709 |      0.0709 |      0.48851 |
|    5 |       6 | Accept |      0.0764 |      46.923 |      0.0709 |    0.070903 |   5.0695e-09 |
|    6 |       6 | Best   |       0.068 |      294.89 |       0.068 |    0.068004 |    0.0029652 |
|    7 |       6 | Accept |       0.125 |      99.095 |       0.068 |    0.068001 |       9.9814 |
|    8 |       6 | Accept |      0.0693 |      321.66 |       0.068 |    0.067999 |    0.0015167 |
|    9 |       6 | Accept |      0.0882 |      138.03 |       0.068 |       0.068 |       1.8203 |
|   10 |       6 | Accept |      0.0753 |      285.07 |       0.068 |    0.067991 |   0.00042423 |
|   11 |       6 | Accept |      0.0764 |      47.704 |       0.068 |    0.067984 |   1.6326e-07 |
|   12 |       6 | Accept |      0.0763 |      46.514 |       0.068 |     0.06798 |   1.0048e-09 |
|   13 |       6 | Best   |      0.0643 |       252.2 |      0.0643 |      0.0643 |     0.095965 |
|   14 |       6 | Accept |      0.0766 |      168.37 |      0.0643 |      0.0643 |   9.1336e-07 |
|   15 |       6 | Accept |      0.0753 |      153.29 |      0.0643 |    0.064301 |   4.8641e-05 |
|   16 |       6 | Accept |      0.0662 |      256.65 |      0.0643 |    0.064298 |    0.0093576 |
|   17 |       6 | Best   |      0.0632 |       224.2 |      0.0632 |    0.063226 |     0.031314 |
|   18 |       6 | Accept |      0.0673 |      219.59 |      0.0632 |    0.063201 |      0.20528 |
|   19 |       6 | Accept |      0.0637 |      244.17 |      0.0632 |    0.063208 |     0.075001 |
|   20 |       6 | Accept |       0.064 |      234.85 |      0.0632 |     0.06321 |     0.081232 |
|================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |       Lambda |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |
|================================================================================================|
|   21 |       6 | Accept |      0.0646 |       242.2 |      0.0632 |    0.063315 |     0.078081 |
|   22 |       6 | Accept |      0.0633 |      217.97 |      0.0632 |    0.063233 |     0.039495 |
|   23 |       6 | Accept |      0.0643 |      224.22 |      0.0632 |    0.063496 |     0.052107 |
|   24 |       6 | Accept |      0.0761 |      45.102 |      0.0632 |    0.063509 |   4.3946e-08 |
|   25 |       6 | Accept |      0.0645 |      221.24 |      0.0632 |    0.063778 |     0.044455 |
|   26 |       6 | Accept |      0.0763 |      44.572 |      0.0632 |    0.063778 |   1.9139e-09 |
|   27 |       6 | Accept |      0.0639 |       216.9 |      0.0632 |    0.063791 |     0.041759 |
|   28 |       6 | Accept |      0.0766 |      45.609 |      0.0632 |     0.06379 |   2.0642e-08 |
|   29 |       6 | Accept |      0.0765 |      121.35 |      0.0632 |    0.063789 |   3.5882e-07 |
|   30 |       6 | Accept |      0.0636 |      215.47 |      0.0632 |    0.063755 |     0.038062 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 952.7987 seconds.
Total objective function evaluation time: 5145.3787

Best observed feasible point:
     Lambda 
    ________

    0.031314

Observed objective function value = 0.0632
Estimated objective function value = 0.063828
Function evaluation time = 224.2018

Best estimated feasible point (according to models):
     Lambda 
    ________

    0.044455

Estimated objective function value = 0.063755
Estimated function evaluation time = 219.4845


Loss5 =

    0.0748

1000 個の特徴量を抽出した RICA に基づく分類器は、200 個の特徴量を抽出した RICA 分類器と同じような検定損失になります。

`bayesopt` の使用によるハイパーパラメーターの最適化

特徴抽出関数には次のような調整パラメーターがあります。

反復制限
関数、rica または sparsefilt
Lambda パラメーター
学習済み特徴量の個数 q

fitcecoc の正則化パラメーターも、学習済み分類器の精度に影響を与えます。このパラメーターもハイパーパラメーターのリストに含めます。

使用可能なパラメーターから効果的に探すため、bayesopt を試します。ワークスペースから渡されるパラメーターが含まれている、次の目的関数を使用します。

function objective = filterica(x,Xtrain,Xtest,LabelTrain,LabelTest,winit)

initW = winit(1:size(Xtrain,2),1:x.q);
if char(x.solver) == 'r'
    Mdl = rica(Xtrain,x.q,'Lambda',x.lambda,'IterationLimit',x.iterlim, ...
        'InitialTransformWeights',initW,'Standardize',true);
else
    Mdl = sparsefilt(Xtrain,x.q,'Lambda',x.lambda,'IterationLimit',x.iterlim, ...
        'InitialTransformWeights',initW);
end

NewX = transform(Mdl,Xtrain);
TestX = transform(Mdl,Xtest);
t = templateLinear('Lambda',x.lambdareg,'Solver','lbfgs');
Cmdl = fitcecoc(NewX,LabelTrain,'Learners',t);
objective = loss(Cmdl,TestX,LabelTest);

変動の原因を取り除くため、初期の変換重み行列を固定します。

W = randn(1e4,1e3);

目的関数のハイパーパラメーターを作成します。

iterlim = optimizableVariable('iterlim',[5,500],'Type','integer');
lambda = optimizableVariable('lambda',[0,10]);
solver = optimizableVariable('solver',{'r','s'},'Type','categorical');
qvar = optimizableVariable('q',[10,1000],'Type','integer');
lambdareg = optimizableVariable('lambdareg',[1e-6,1],'Transform','log');
vars = [iterlim,lambda,solver,qvar,lambdareg];

内部最適化の実行が完了しないときに発生する警告なしで最適化を実行します。最適化で良好な値が見つかる可能性を向上させるため、既定の 30 回ではなく 60 回の反復で実行します。

warning('off','stats:classreg:learning:fsutils:Solver:LBFGSUnableToConverge');
results = bayesopt(@(x) filterica(x,Xtrain,Xtest,LabelTrain,LabelTest,W),vars, ...
    'UseParallel',true,'MaxObjectiveEvaluations',60);
warning('on','stats:classreg:learning:fsutils:Solver:LBFGSUnableToConverge');

Copying objective function to workers...
Done copying objective function to workers.
|============================================================================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |      iterlim |       lambda |       solver |            q |    lambdareg |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |              |              |              |              |
|============================================================================================================================================================|
|    1 |       6 | Best   |     0.16408 |      33.743 |     0.16408 |     0.16408 |          140 |       9.4661 |            s |           98 |    0.0007106 |
|    2 |       6 | Best   |    0.079213 |      51.975 |    0.079213 |     0.09064 |           10 |        9.466 |            r |          685 |     0.010462 |
|    3 |       6 | Best   |    0.074897 |      82.031 |    0.074897 |    0.074983 |           32 |       3.7554 |            r |          689 |      0.13737 |
|    4 |       6 | Accept |     0.07546 |      93.221 |    0.074897 |    0.075073 |          178 |       3.9741 |            r |          196 |       0.1829 |
|    5 |       6 | Accept |     0.13924 |      30.444 |    0.074897 |    0.074933 |          282 |      0.36123 |            r |           33 |      0.99029 |
|    6 |       6 | Accept |    0.083964 |         133 |    0.074897 |    0.074933 |           58 |       9.7653 |            r |          685 |    0.0014623 |
|    7 |       6 | Accept |     0.08128 |      33.609 |    0.074897 |    0.074957 |            8 |       5.6351 |            r |          519 |    0.0065822 |
|    8 |       6 | Accept |    0.090751 |      203.96 |    0.074897 |    0.074913 |          131 |      0.73308 |            r |          577 |   2.1172e-05 |
|    9 |       6 | Accept |    0.090001 |      172.38 |    0.074897 |    0.074904 |          146 |       8.1899 |            r |          454 |   1.4417e-05 |
|   10 |       6 | Accept |    0.080191 |       316.8 |    0.074897 |    0.074897 |          164 |      0.48783 |            r |          727 |     0.004936 |
|   11 |       6 | Best   |    0.060472 |      40.777 |    0.060472 |    0.060731 |            5 |       2.3201 |            s |          530 |   1.1957e-06 |
|   12 |       6 | Accept |    0.079027 |      45.841 |    0.060472 |    0.060632 |            8 |      0.55541 |            r |          696 |     0.030914 |
|   13 |       6 | Accept |    0.074823 |      237.43 |    0.060472 |     0.06067 |          109 |       4.5352 |            r |          781 |      0.12274 |
|   14 |       6 | Accept |     0.84009 |      85.121 |    0.060472 |    0.060468 |          306 |      0.59533 |            s |          148 |      0.89675 |
|   15 |       6 | Accept |     0.15637 |      200.13 |    0.060472 |    0.060451 |           90 |       3.0192 |            s |          999 |    0.0043768 |
|   16 |       6 | Accept |     0.69006 |      14.273 |    0.060472 |     0.06047 |            6 |       9.4568 |            s |          407 |      0.13833 |
|   17 |       6 | Accept |    0.093035 |      205.83 |    0.060472 |    0.060469 |          263 |       2.3083 |            r |          308 |   1.0016e-06 |
|   18 |       6 | Accept |     0.18753 |      6.0238 |    0.060472 |    0.060527 |           36 |        9.806 |            s |           24 |   8.3653e-06 |
|   19 |       6 | Accept |       0.119 |      749.98 |    0.060472 |    0.060751 |          482 |      0.51927 |            s |          818 |   1.5416e-06 |
|   20 |       6 | Accept |    0.076414 |      751.21 |    0.060472 |    0.060754 |          387 |       9.9936 |            r |          784 |      0.26786 |
|============================================================================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |      iterlim |       lambda |       solver |            q |    lambdareg |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |              |              |              |              |
|============================================================================================================================================================|
|   21 |       6 | Accept |    0.099332 |      7.2298 |    0.060472 |    0.060828 |           20 |      0.78894 |            s |           49 |   1.0335e-06 |
|   22 |       6 | Accept |    0.090139 |      7.9815 |    0.060472 |    0.060858 |           11 |       3.2973 |            r |           88 |   2.7437e-06 |
|   23 |       6 | Accept |    0.076696 |      323.64 |    0.060472 |    0.060872 |          120 |       1.9199 |            r |          999 |       0.2537 |
|   24 |       6 | Accept |    0.098003 |      50.544 |    0.060472 |    0.060876 |          492 |       1.7197 |            r |           27 |   0.00020896 |
|   25 |       6 | Accept |     0.10383 |      56.568 |    0.060472 |     0.06101 |           11 |        5.256 |            s |          971 |   0.00054471 |
|   26 |       6 | Accept |     0.14405 |      30.426 |    0.060472 |    0.060797 |          477 |       5.5475 |            r |           12 |     0.022342 |
|   27 |       6 | Accept |     0.09046 |      53.398 |    0.060472 |    0.060815 |           13 |       2.1216 |            r |          986 |   1.1811e-06 |
|   28 |       6 | Best   |    0.051641 |      99.452 |    0.051641 |    0.051368 |           23 |       2.6976 |            s |          985 |   1.0558e-06 |
|   29 |       6 | Accept |     0.10016 |      6.4162 |    0.051641 |    0.051365 |            6 |       3.7223 |            r |           69 |   9.2926e-05 |
|   30 |       6 | Accept |     0.10943 |      40.676 |    0.051641 |    0.051391 |          488 |       5.2092 |            r |           19 |   2.4162e-05 |
|   31 |       6 | Accept |    0.086761 |      7.8419 |    0.051641 |    0.051393 |           24 |       6.5535 |            r |           42 |    0.0013244 |
|   32 |       6 | Best   |      0.0504 |      96.816 |      0.0504 |    0.050526 |           14 |        9.929 |            s |         1000 |   2.8809e-06 |
|   33 |       6 | Accept |    0.088789 |      81.158 |      0.0504 |    0.050525 |           14 |       1.0441 |            r |          927 |   0.00021061 |
|   34 |       6 | Accept |    0.083083 |      887.17 |      0.0504 |     0.05052 |          351 |       6.8834 |            r |          978 |    0.0026404 |
|   35 |       6 | Best   |    0.050023 |      99.493 |    0.050023 |    0.050372 |           19 |       9.9813 |            s |          899 |   1.0257e-06 |
|   36 |       6 | Accept |    0.053338 |      113.36 |    0.050023 |    0.050499 |            7 |       4.7855 |            s |          984 |   1.8611e-06 |
|   37 |       6 | Accept |    0.089024 |      70.047 |    0.050023 |      0.0505 |           15 |       8.8301 |            r |          984 |   6.0636e-06 |
|   38 |       6 | Accept |    0.052029 |      95.822 |    0.050023 |    0.050551 |            7 |        9.759 |            s |          996 |   3.7871e-06 |
|   39 |       6 | Accept |    0.085992 |      73.422 |    0.050023 |    0.050528 |            5 |       2.7837 |            r |          968 |     0.004483 |
|   40 |       6 | Accept |    0.091159 |      5.8348 |    0.050023 |     0.05052 |           15 |       8.7732 |            r |           37 |     0.084632 |
|============================================================================================================================================================|
| Iter | Active  | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   |      iterlim |       lambda |       solver |            q |    lambdareg |
|      | workers | result |             | runtime     | (observed)  | (estim.)    |              |              |              |              |              |
|============================================================================================================================================================|
|   41 |       6 | Best   |    0.046444 |      152.93 |    0.046444 |    0.047062 |           30 |       4.0843 |            s |          997 |    7.279e-06 |
|   42 |       6 | Accept |    0.052712 |      58.107 |    0.046444 |     0.04698 |           12 |      0.99592 |            s |          652 |   1.0258e-06 |
|   43 |       6 | Accept |    0.058005 |      91.928 |    0.046444 |    0.047263 |           10 |        5.511 |            s |         1000 |   2.4589e-05 |
|   44 |       6 | Accept |    0.055413 |      103.25 |    0.046444 |    0.047306 |            7 |       5.6791 |            s |          953 |   1.4656e-06 |
|   45 |       6 | Accept |    0.052517 |      96.201 |    0.046444 |    0.049604 |           10 |       5.9403 |            s |          996 |   1.0525e-05 |
|   46 |       6 | Accept |    0.089527 |      76.617 |    0.046444 |    0.046888 |           20 |       1.0744 |            r |          965 |      0.96766 |
|   47 |       6 | Accept |    0.050062 |      99.709 |    0.046444 |    0.046735 |           12 |       9.9236 |            s |          975 |   4.5916e-06 |
|   48 |       6 | Accept |     0.21166 |      90.117 |    0.046444 |    0.049716 |          495 |       1.1996 |            s |           86 |   0.00022338 |
|   49 |       6 | Accept |    0.054535 |        79.1 |    0.046444 |    0.046679 |            6 |      0.22929 |            s |          967 |   7.6974e-06 |
|   50 |       6 | Accept |     0.12385 |      964.74 |    0.046444 |    0.049963 |          474 |       4.7085 |            s |          991 |   8.6984e-05 |
|   51 |       6 | Accept |    0.052016 |      76.098 |    0.046444 |    0.049914 |           10 |       1.0798 |            s |          922 |    1.133e-06 |
|   52 |       6 | Accept |    0.048984 |      95.054 |    0.046444 |    0.049891 |           12 |         4.69 |            s |          976 |   1.0189e-06 |
|   53 |       6 | Accept |      0.1948 |      889.11 |    0.046444 |    0.047903 |          466 |       7.9582 |            s |          986 |    0.0012319 |
|   54 |       6 | Accept |     0.10652 |       5.076 |    0.046444 |    0.047961 |           10 |       5.9107 |            r |           40 |      0.52677 |
|   55 |       6 | Accept |    0.074194 |      319.41 |    0.046444 |     0.04981 |          130 |       2.6437 |            s |          997 |   7.8756e-06 |
|   56 |       6 | Accept |      0.1014 |      45.184 |    0.046444 |    0.049828 |          480 |       6.1835 |            r |           24 |   2.0019e-06 |
|   57 |       6 | Accept |     0.33214 |      3.1996 |    0.046444 |    0.049785 |           12 |       7.4538 |            s |           13 |     0.016248 |
|   58 |       6 | Accept |    0.054348 |      96.616 |    0.046444 |    0.050832 |           12 |       2.8605 |            s |          987 |   4.7951e-06 |
|   59 |       6 | Accept |     0.71471 |      3.0555 |    0.046444 |    0.050852 |           10 |       9.8909 |            s |           24 |      0.21362 |
|   60 |       6 | Accept |    0.074353 |      67.118 |    0.046444 |     0.05084 |            8 |       5.5275 |            s |          986 |   8.9716e-05 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 60 reached.
Total function evaluations: 60
Total elapsed time: 1921.1117 seconds.
Total objective function evaluation time: 9107.7006

Best observed feasible point:
    iterlim    lambda    solver     q     lambdareg
    _______    ______    ______    ___    _________

      30       4.0843      s       997    7.279e-06

Observed objective function value = 0.046444
Estimated objective function value = 0.053743
Function evaluation time = 152.932

Best estimated feasible point (according to models):
    iterlim    lambda    solver     q     lambdareg
    _______    ______    ______    ___    _________

      10       1.0798      s       922    1.133e-06

Estimated objective function value = 0.05084
Estimated function evaluation time = 90.9315

生成された分類器では、1000 個の特徴量で sparsefilt を使用した分類器に 10 回の反復で学習をさせた場合より、損失が改善されては (小さくなっては) いません。

bayesopt で求められた最適なハイパーパラメーターについてフィルターの係数を確認します。生成されたイメージは、抽出された特徴量の形状を示しています。これらの形状は、手書き数字の一部として認識できます。

Xtbl = results.XAtMinObjective;
Q = Xtbl.q;
initW = W(1:size(Xtrain,2),1:Q);
if char(Xtbl.solver) == 'r'
    Mdl = rica(Xtrain,Q,'Lambda',Xtbl.lambda,'IterationLimit',Xtbl.iterlim, ...
        'InitialTransformWeights',initW,'Standardize',true);
else
    Mdl = sparsefilt(Xtrain,Q,'Lambda',Xtbl.lambda,'IterationLimit',Xtbl.iterlim, ...
        'InitialTransformWeights',initW);
end
Wts = Mdl.TransformWeights;
Wts = reshape(Wts,[28,28,Q]);
[dx,dy,~,~] = size(Wts);
for f = 1:Q
    Wvec = Wts(:,:,f);
    Wvec = Wvec(:);
    Wvec =(Wvec - min(Wvec))/(max(Wvec) - min(Wvec));
    Wts(:,:,f) = reshape(Wvec,dx,dy);
end
m   = ceil(sqrt(Q));
n   = m;
img = zeros(m*dx,n*dy);
f   = 1;
for i = 1:m
    for j = 1:n
        if (f <= Q)
            img((i-1)*dx+1:i*dx,(j-1)*dy+1:j*dy,:) = Wts(:,:,f);
            f = f+1;
        end
    end
end
imshow(img);

Warning: Solver LBFGS was not able to converge to a solution.

MNIST データを読み込むためのコード

データをワークスペースに読み込む関数のコードは以下のとおりです。

function [X,L] = processMNISTdata(imageFileName,labelFileName)

[fileID,errmsg] = fopen(imageFileName,'r','b');
if fileID < 0
    error(errmsg);
end
%%
% First read the magic number. This number is 2051 for image data, and
% 2049 for label data
magicNum = fread(fileID,1,'int32',0,'b');
if magicNum == 2051
    fprintf('\nRead MNIST image data...\n')
end
%%
% Then read the number of images, number of rows, and number of columns
numImages = fread(fileID,1,'int32',0,'b');
fprintf('Number of images in the dataset: %6d ...\n',numImages);
numRows = fread(fileID,1,'int32',0,'b');
numCols = fread(fileID,1,'int32',0,'b');
fprintf('Each image is of %2d by %2d pixels...\n',numRows,numCols);
%%
% Read the image data
X = fread(fileID,inf,'unsigned char');
%%
% Reshape the data to array X
X = reshape(X,numCols,numRows,numImages);
X = permute(X,[2 1 3]);
%%
% Then flatten each image data into a 1 by (numRows*numCols) vector, and 
% store all the image data into a numImages by (numRows*numCols) array.
X = reshape(X,numRows*numCols,numImages)';
fprintf(['The image data is read to a matrix of dimensions: %6d by %4d...\n',...
    'End of reading image data.\n'],size(X,1),size(X,2));
%%
% Close the file
fclose(fileID);
%%
% Similarly, read the label data.
[fileID,errmsg] = fopen(labelFileName,'r','b');
if fileID < 0
    error(errmsg);
end
magicNum = fread(fileID,1,'int32',0,'b');
if magicNum == 2049
    fprintf('\nRead MNIST label data...\n')
end
numItems = fread(fileID,1,'int32',0,'b');
fprintf('Number of labels in the dataset: %6d ...\n',numItems);

L = fread(fileID,inf,'unsigned char');
fprintf(['The label data is read to a matrix of dimensions: %6d by %2d...\n',...
    'End of reading label data.\n'],size(L,1),size(L,2));
fclose(fileID);

参考文献

[1] 元の NIST データセットから派生して作成された MNIST データセットの著作権は Yann LeCun (Courant Institute, NYU) と Corinna Cortes (Google Labs, New York) が有しています。MNIST データセットは、Creative Commons Attribution-Share Alike 3.0 ライセンス (https://creativecommons.org/licenses/by-sa/3.0/) の条件下で使用可能になります。