Ouch

[sandermvdb]

I have 2 versions of my engine installed, one stable version and one 'experimental' version. This way I verified that I don't have any transposition table bugs by starting a small tournament where one engine has 1 mb TT and the other has 128 and calculate till a depth of 11. Both were about equal so no bug there.
Lately I have tried to tweak my LMR, static nullmove pruning, fixed a bug in SEE but all tweaks seemed to weaken my engine. Felt I had hit a wall (which I probably will sometime)...
I also just simplified my king-safety logic and to verify the same scores were calculated I again started a tournament. But again my experimental version was weaker. I verified a lot of evaluation scores at different positions but these were the same. What is going on?!
Then for some reason, I opened the windows taskbar and saw processes with +/- 160 mb and processes with +/- 40 mb. I had found the cause!! My experimental version was still only using 1 mb for the transposition table
Too bad the testresults were flawed but I guess I have not yet hit the wall!

[hgm]

Your engine uses 40MB for things other than the transposition table?

[sandermvdb]

Your engine uses 40MB for things other than the transposition table?

No, it's the JVM that is assigning that much memory. Other things are a couple of mb, max.

[stegemma]

Your engine uses 40MB for things other than the transposition table?

No, it's the JVM that is assigning that much memory. Other things are a couple of mb, max.

So maybe your wall is the JVM!

[sandermvdb]

Your engine uses 40MB for things other than the transposition table?

No, it's the JVM that is assigning that much memory. Other things are a couple of mb, max.

So maybe your wall is the JVM!

That would be an easy excuse, but I know I have to look elsewhere

[mar]

That would be an easy excuse, but I know I have to look elsewhere

Actually that's not an excuse, Java is pile of crap, literally.

No structures, no control over what goes to stack except for elementary types.
"Everything is an object" is road to hell (even if it makes VM programmers' life a LOT easier along with GC, but the price for that is to be paid later - which is exactly what you've experienced).
Even with object pools (which defeat nice syntactic sugar), it's still horrible - especially for people who lack understanding of what's going on under the hood.
Every object is god knows where in ram so performance goes to hell and when GC triggers its performance is inversely proportional to # of live objects (=everything in Java).

So even if their VM generates state of art code, it'll still suffer because of that.

C# is just an attempt to "Java done right", D is AOT instead of JIT so those are better alternatives.

Still I would prefer a "non-managed" (read non-GC) language over those anytime - because I have full control and I like that,
there are better alternatives to optimizing your program other than buying more ram and/or tweaking VM params

[Ras]

Your engine uses 40MB for things other than the transposition table?

Feeling funny reading things like this - with system RAM of 192k.

[sandermvdb]

That would be an easy excuse, but I know I have to look elsewhere

Actually that's not an excuse, Java is pile of crap, literally.

No structures, no control over what goes to stack except for elementary types.
"Everything is an object" is road to hell (even if it makes VM programmers' life a LOT easier along with GC, but the price for that is to be paid later - which is exactly what you've experienced).
Even with object pools (which defeat nice syntactic sugar), it's still horrible - especially for people who lack understanding of what's going on under the hood.
Every object is god knows where in ram so performance goes to hell and when GC triggers its performance is inversely proportional to # of live objects (=everything in Java).

So even if their VM generates state of art code, it'll still suffer because of that.

C# is just an attempt to "Java done right", D is AOT instead of JIT so those are better alternatives.

Still I would prefer a "non-managed" (read non-GC) language over those anytime - because I have full control and I like that,
there are better alternatives to optimizing your program other than buying more ram and/or tweaking VM params

I disagree with most about what you are saying but don't feel like arguing about this.
Carballo which is also written in Java has an elo of 2700+, so plenty of work to do

[brtzsnr]

For most chess engines doubling the speed only gives you 30-50 Elo. For example 4 CPU version is only 60-100 Elo stronger than the 1 CPU version for most engines. C / C++ / D are only 10-25% faster than Java / Golang on most CPU intensive tasks so the Elo gain you get from being closer to the metal is even smaller. OTOH, the programmer's productivity increase of these languages make up for it quite well, even more than the lost performance. For example, I've never had to debug any memory issue on my engine, any out of bounds error was immediately caught with a nice stack trace of the origin.

[sandermvdb]

Some extra info:
If I let my engine search for a couple of minutes with 128mb TT, I get one garbage collection which takes 9 msec.
If I set the TT to 1 mb, I can force the JVM to only take 3 mb of RAM and just tested that is then just as fast as when using +/- 40 mb.