lukasmonk wrote: ↑Sun Jun 23, 2019 12:10 pm
An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.
Isn't it easier to allow the program to access windows virtual memory? That way it only uses the disk when out of physical memory.
Yes, you're probably right, but windows without memory is not very stable, at least in my experience, and on the other hand, using sqlite for example, is very slow, but you can work in the meantime on any program, with almost all resources.
The slowness generating the book can be a problem, maybe it is excessive, surely there are better intermediate solutions.
lukasmonk wrote: ↑Sun Jun 23, 2019 12:10 pm
An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.
Isn't it easier to allow the program to access windows virtual memory? That way it only uses the disk when out of physical memory.
Yes, you're probably right, but windows without memory is not very stable, at least in my experience, and on the other hand, using sqlite for example, is very slow, but you can work in the meantime on any program, with almost all resources.
The slowness generating the book can be a problem, maybe it is excessive, surely there are better intermediate solutions.
YMMV of course, but I never had problems letting the system starve completely and work with the virtual memory. It was slower, but not that much with a fast NVMe drive.
lukasmonk wrote: ↑Sun Jun 23, 2019 12:10 pm
An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.
Isn't it easier to allow the program to access windows virtual memory? That way it only uses the disk when out of physical memory.
Yes, you're probably right, but windows without memory is not very stable, at least in my experience, and on the other hand, using sqlite for example, is very slow, but you can work in the meantime on any program, with almost all resources.
The slowness generating the book can be a problem, maybe it is excessive, surely there are better intermediate solutions.
Sqlite can use in memory tables. https://www.sqlite.org/inmemorydb.html
Also, Fastdb is an in memory database, as is Monetdb.
Monetdb is the most sophisticated of these, but it is not an embedded database
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Dann Corbit wrote: ↑Tue Jun 25, 2019 8:27 pm
Sqlite can use in memory tables. https://www.sqlite.org/inmemorydb.html
Also, Fastdb is an in memory database, as is Monetdb.
Monetdb is the most sophisticated of these, but it is not an embedded database
I have tried two options with sqlite, one using the memory mode, another using a file but with cache, and the conclusions I have obtained are that the memory mode has the same problem as poliglot.exe, in terms of the increasing use of the memory, and the difference in time with respect to the file mode is minimal.
The file mode has the advantage of a stable memory in low values (determined by the size of the cache), but takes 8 to 10 times more in time (in relation to the official poliglot.exe in windows).
