So far I found that to learn only 4000 training examples I already need about 29 layers with 4 filters per layer in an ordinary convolutional neural network.
To fit more examples I expect more layers and filters. Probably I need to switch to residual network too when number of layers increases.
Overfitting should be prevented. But this time I'm only focussing on training error.
The more layers and filters the slower it gets.
Minimal network
Moderators: hgm, Rebel, chrisw
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: Minimal network
To get a lower validation error ( I mean lower than first epoch) I needed at least about 3000 training examples in my ultra minimal network of two conv layers where each layer has only one filter. For instance 2000 training examples did not give a lower validation error. Maybe if you add noise you need less examples.
Cool information. Isn't it.
By the way minimum validation for 3000 examples is about 0.302 for this network.
Of course no progress. I am not obsessed by movement.
Hmm then what about first epoch. How many training examples are needed to get any better validation error when using only one epoch. To be continued
Cool information. Isn't it.
Code: Select all
0.304705596535846 0.299141061043657
0.304200726404863 0.296937026701584
0.303885715964357 0.29536382114373
0.303647085111191 0.294051130754107
0.303474092302082 0.292888186599641
0.303335064270266 0.291813712011117
0.303245155439217 0.290795499021344
By the way minimum validation for 3000 examples is about 0.302 for this network.
Of course no progress. I am not obsessed by movement.
Hmm then what about first epoch. How many training examples are needed to get any better validation error when using only one epoch. To be continued
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: Minimal network
Using only one training example gives a validation error of 0.305. Probably result depends on quality of weight initialization.
Here an error of 0.305 is equivalent to an error of about a rook (450 CP)
Here an error of 0.305 is equivalent to an error of about a rook (450 CP)
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: Minimal network
Minimum validation error of two layer conv using two filters per layer and 200K training examples appears to be about 0.153 in my network.
That would be equal to 240 centipawn (half rook, or bad bishop). So useless.
Increasing number of training examples did not help because 100K training examples gave 0.154
That would be equal to 240 centipawn (half rook, or bad bishop). So useless.
Increasing number of training examples did not help because 100K training examples gave 0.154
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: Minimal network
Validation error four layers one filter each does not go below val 0.2 using 100K training examples
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Minimal network
Isn't the minimal network a single perceptron ? Most hand-written chess evaluation can be represented a perceptron with nHenk wrote: ↑Mon Oct 01, 2018 10:29 am So far I found that to learn only 4000 training examples I already need about 29 layers with 4 filters per layer in an ordinary convolutional neural network.
To fit more examples I expect more layers and filters. Probably I need to switch to residual network too when number of layers increases.
Overfitting should be prevented. But this time I'm only focussing on training error.
The more layers and filters the slower it gets.
inputs and a single output for the evaluation (0 to 1).
I think you should pick a specific accuracy (say 70%) to define the minimum network size that would achieve it.
The validation set should be sperate from training set so that one could do several epochs to overfit the training set.
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: Minimal network
3 filters did no go below 0.13
I skip 5 filters so trying now 7 filters. Each layer or trap uses a copy layer(connection) too. Like in residual network. I'm currently using only eight layers so residual network not necessary.
Downside is that network gets slower and slower.
-
- Posts: 2663
- Joined: Wed Mar 10, 2010 10:18 pm
- Location: Hamburg, Germany
- Full name: Srdja Matovic
Re: Minimal network
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: Minimal network
Yes i know they are not that expensive but I don't know how to install them in my current laptop. Actually I'm scared of opening computers. Maybe afterwards my laptop won't work anymore etc. Maybe if there is some fool proof method to install them. I don't know yet.smatovic wrote: ↑Thu Oct 18, 2018 5:08 pm1 TFLOPS for 50 bucks...
https://www.ebay.com/sch/i.html?_nkw=nvidia+GTX+750
--
Srdja