Don't use databases

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Henk
Posts: 7215
Joined: Mon May 27, 2013 10:31 am

Don't use databases

Post by Henk »

After training a network it saves network weights in a database. But once in a while it fails. So all tuning was for nothing for new weights weren't stored.

Message: Test method Skipper3.Tests.NetworkTest.TrainNetwork threw exception:
System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> System.Data.SqlClient.SqlException: Cannot open database "Skipper3" requested by the login. The login failed.
Login failed for user 'hp\Henk'.
User avatar
lucasart
Posts: 3232
Joined: Mon May 31, 2010 1:29 pm
Full name: lucasart

Re: Don't use databases

Post by lucasart »

Henk wrote: Sat Oct 13, 2018 12:30 pm After training a network it saves network weights in a database. But once in a while it fails. So all tuning was for nothing for new weights weren't stored.

Message: Test method Skipper3.Tests.NetworkTest.TrainNetwork threw exception:
System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> System.Data.SqlClient.SqlException: Cannot open database "Skipper3" requested by the login. The login failed.
Login failed for user 'hp\Henk'.
Thank you for this insightful post.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
odomobo
Posts: 96
Joined: Fri Jul 06, 2018 1:09 am
Location: Chicago, IL
Full name: Josh Odom

Re: Don't use databases

Post by odomobo »

Maybe it's better using sqlite, as long as you aren't using any advanced DB features. It doesn't have permissions, apart from standard file permissions.
Henk
Posts: 7215
Joined: Mon May 27, 2013 10:31 am

Re: Don't use databases

Post by Henk »

"Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was "



I had to wait at least five minutes before data connection to database could be used.
Maybe because my computer was too busy.

Why don't they give a warning if you want to install SQL server on a slow computer.

I don't know what is going on.

Might be windows 10 secret services being overly busy. I don't know.

Or maybe this database contains too many chess positions? Makes no sense too.
Dann Corbit
Posts: 12536
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Don't use databases

Post by Dann Corbit »

Henk wrote: Sat Oct 13, 2018 12:30 pm After training a network it saves network weights in a database. But once in a while it fails. So all tuning was for nothing for new weights weren't stored.

Message: Test method Skipper3.Tests.NetworkTest.TrainNetwork threw exception:
System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> System.Data.SqlClient.SqlException: Cannot open database "Skipper3" requested by the login. The login failed.
Login failed for user 'hp\Henk'.
A database, when used properly, is more reliable than a file, not less reliable.
A database will give you a reliable guarantee that the data was saved.
A database can give you time ordered generations of your work.
A database has many more advantages.
A database can save floating point numbers in native floating point format and retrieve them very quickly.
If you use a memory mapped database like FastDB or Monetdb, you have an ultra fast memory mapped file at your disposal, with all the reliability kinks worked out.

When your login failed, there are basically just a few possibilities of what went wrong:
1. You gave the wrong user name
2. You gave a wrong password
3. You used integrated security and did not give rights to the user who was connecting
4. The DLL used for performing the queries was not in scope or was the wrong "bitness" (e.g. you have a 32 bit DLL but your app is 64 bits)

But of course, a plain file will also work. On the other hand, many of these things can go wrong when you are trying to use a file:
http://pubs.opengroup.org/onlinepubs/00 ... rno.h.html
So it can be more problematic than you might imagine.
E.g. do you have permissions to the folder?
E.g. the file is locked by another user
E.g. the disk is full

A database has a nice way to tell you what is wrong.
With an ordinary file, you will have to do some operating system specific lookups, which are hard to make portable.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Dann Corbit
Posts: 12536
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Don't use databases

Post by Dann Corbit »

Henk wrote: Fri Nov 09, 2018 3:34 pm "Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was "



I had to wait at least five minutes before data connection to database could be used.
Maybe because my computer was too busy.

Why don't they give a warning if you want to install SQL server on a slow computer.

I don't know what is going on.

Might be windows 10 secret services being overly busy. I don't know.

Or maybe this database contains too many chess positions? Makes no sense too.
SQL Server (when installed with defaults) makes lots of assumptions about resources that are not necessarily valid.
One of them is "all your resources are belong to us!", since the SS team is probably run by aliens.
SQL Server will use all the available memory. Then when you run some other task that consumes a lot of memory, you have paging madness.
So limit SQL Server to a sensible amount of memory (under: server properties/memory/maximum memory). Also, don't install SS on a wimpy machine unless you want wimpy data access.
The suggestion of using an embedded database like SQLite is a good one. An embedded database tends to be quite fast, but the down side is it is normally limited to a single user. If that is OK for your application, it is often a good idea.

Once you get used to doing database access, you will never want to go back to files.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Henk
Posts: 7215
Joined: Mon May 27, 2013 10:31 am

Re: Don't use databases

Post by Henk »

Dann Corbit wrote: Fri Nov 09, 2018 9:54 pm
Henk wrote: Fri Nov 09, 2018 3:34 pm "Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was "



I had to wait at least five minutes before data connection to database could be used.
Maybe because my computer was too busy.

Why don't they give a warning if you want to install SQL server on a slow computer.

I don't know what is going on.

Might be windows 10 secret services being overly busy. I don't know.

Or maybe this database contains too many chess positions? Makes no sense too.
SQL Server (when installed with defaults) makes lots of assumptions about resources that are not necessarily valid.
One of them is "all your resources are belong to us!", since the SS team is probably run by aliens.
SQL Server will use all the available memory. Then when you run some other task that consumes a lot of memory, you have paging madness.
So limit SQL Server to a sensible amount of memory (under: server properties/memory/maximum memory). Also, don't install SS on a wimpy machine unless you want wimpy data access.
The suggestion of using an embedded database like SQLite is a good one. An embedded database tends to be quite fast, but the down side is it is normally limited to a single user. If that is OK for your application, it is often a good idea.

Once you get used to doing database access, you will never want to go back to files.
Disk on my computer is sometimes overly busy. System, viruschecker, visual studio are using it too much. I can close visual studio. Or maybe I'm using visual studio wrongly for I forgot to close a great many open files. Don't know yet if that is the main cause.
Henk
Posts: 7215
Joined: Mon May 27, 2013 10:31 am

Re: Don't use databases

Post by Henk »

Henk wrote: Mon Nov 12, 2018 10:46 am
Dann Corbit wrote: Fri Nov 09, 2018 9:54 pm
Henk wrote: Fri Nov 09, 2018 3:34 pm "Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was "



I had to wait at least five minutes before data connection to database could be used.
Maybe because my computer was too busy.

Why don't they give a warning if you want to install SQL server on a slow computer.

I don't know what is going on.

Might be windows 10 secret services being overly busy. I don't know.

Or maybe this database contains too many chess positions? Makes no sense too.
SQL Server (when installed with defaults) makes lots of assumptions about resources that are not necessarily valid.
One of them is "all your resources are belong to us!", since the SS team is probably run by aliens.
SQL Server will use all the available memory. Then when you run some other task that consumes a lot of memory, you have paging madness.
So limit SQL Server to a sensible amount of memory (under: server properties/memory/maximum memory). Also, don't install SS on a wimpy machine unless you want wimpy data access.
The suggestion of using an embedded database like SQLite is a good one. An embedded database tends to be quite fast, but the down side is it is normally limited to a single user. If that is OK for your application, it is often a good idea.

Once you get used to doing database access, you will never want to go back to files.
Disk on my computer is sometimes overly busy. System, viruschecker, visual studio are using it too much. I can close visual studio. Or maybe I'm using visual studio wrongly for I forgot to close a great many open files. Don't know yet if that is the main cause.
Might be anti virus checker or system is responsible as well. What should I do to speed them up. Remove 80% of my files. So there is nothing to check. Change operating system? Is there an easier solution. Disk is often utilized for 100%. But even when it is not utilized it is slow. So perhaps disconnect from internet. I don't like that too. Trapped. Now after an hour my computer is more responsive. Otherwise I would not even be able to type this message.

So the situation is that after I start my computer in the morning I have at least one hour problems with response of my computer before I can use it.


My visual studio using now 500MB of memory. What idiocy is that. What did I do wrong. scroll bars of my browser are now responding slow too.
Henk
Posts: 7215
Joined: Mon May 27, 2013 10:31 am

Re: Don't use databases

Post by Henk »

Henk wrote: Fri Nov 16, 2018 1:13 pm
Henk wrote: Mon Nov 12, 2018 10:46 am
Dann Corbit wrote: Fri Nov 09, 2018 9:54 pm
Henk wrote: Fri Nov 09, 2018 3:34 pm "Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was "



I had to wait at least five minutes before data connection to database could be used.
Maybe because my computer was too busy.

Why don't they give a warning if you want to install SQL server on a slow computer.

I don't know what is going on.

Might be windows 10 secret services being overly busy. I don't know.

Or maybe this database contains too many chess positions? Makes no sense too.
SQL Server (when installed with defaults) makes lots of assumptions about resources that are not necessarily valid.
One of them is "all your resources are belong to us!", since the SS team is probably run by aliens.
SQL Server will use all the available memory. Then when you run some other task that consumes a lot of memory, you have paging madness.
So limit SQL Server to a sensible amount of memory (under: server properties/memory/maximum memory). Also, don't install SS on a wimpy machine unless you want wimpy data access.
The suggestion of using an embedded database like SQLite is a good one. An embedded database tends to be quite fast, but the down side is it is normally limited to a single user. If that is OK for your application, it is often a good idea.

Once you get used to doing database access, you will never want to go back to files.
Disk on my computer is sometimes overly busy. System, viruschecker, visual studio are using it too much. I can close visual studio. Or maybe I'm using visual studio wrongly for I forgot to close a great many open files. Don't know yet if that is the main cause.
Might be anti virus checker or system is responsible as well. What should I do to speed them up. Remove 80% of my files. So there is nothing to check. Change operating system? Is there an easier solution. Disk is often utilized for 100%. But even when it is not utilized it is slow. So perhaps disconnect from internet. I don't like that too. Trapped. Now after an hour my computer is more responsive. Otherwise I would not even be able to type this message.

So the situation is that after I start my computer in the morning I have at least one hour problems with response of my computer before I can use it.


My visual studio using now 500MB of memory. What idiocy is that. What did I do wrong. scroll bars of my browser are now responding slow too.
So the only way to use visual studio 2017 for my chess engine project is to close it.
Even saw 700MB of memory usage. Sql server is on second place.

Same holds for web browsers. They have trouble with processing images.
Get lost with your stupid images wasting resources.
Henk
Posts: 7215
Joined: Mon May 27, 2013 10:31 am

Re: Don't use databases

Post by Henk »

Message: Test method Skipper3.Tests.NetworkTest.TrainNetwork threw exception:
System.Data.SqlClient.SqlException: Er is een netwerkfout of een exemplaarspecifieke fout opgetreden tijdens het maken van verbinding met SQL Server. De server is niet gevonden of is niet toegankelijk. Controleer of de exemplaarnaam correct is en of in de instellingen van SQL Server externe verbindingen zijn toegestaan. (provider: SQL Network Interfaces, error: 50 - Er is een fout in de Local Database Runtime opgetreden. Error occurred during LocalDB instance startup: SQL Server process failed to start.
)


Maybe I should put "Integrated Security=False" in connection string I don't know.
Old value was "Integrated Security=SSPI"

https://stackoverflow.com/questions/122 ... ed-securit

Sqlserver complicated?