Weird disc behaviour

[Rebel]

I have this simple example code, copy file, 40 bytes per position.

Code: Select all

        fp1 = fopen(bin_one, "rb");
        fp2 = fopen(bin_two, "rb");

next:   x=fread(&k1,8,1,fp1); if (x<1) goto done;
        fread(&k2,8,1,fp1);
        fread(&k3,8,1,fp1);
        fread(&k4,8,1,fp1);
        fread(&k5,8,1,fp1);

        fwrite(&k1,8,1,fp2); 
        fwrite(&k2,8,1,fp2); 
        fwrite(&k3,8,1,fp2); 
        fwrite(&k4,8,1,fp2); 
        fwrite(&k5,8,1,fp2); 

        goto next;

done:   fclose(fp1);
        fclose(fp2);

This goes well the first 2 billion positions, average speed 6 million per second, then it starts to stall, average speed eventually goes down to less than 1 million per second.

Why is that?

Windows ?

[hgm]

OS disk caching? Initially the actual writing is postponed, giving priority to the reading requests that are always there. Eventually the disk cash is entirely filled up with dirty blocks, and nothing more can be read without first writing one of the blocks to disk and releasing the memory it was occupying. From that moment the HD alternates reads and writes, so that access is no longer sequential, and you get a lot of seek and rotational delay.

It might help to explicitly buffer large chunks of data, and write those out without any reading in between.

[Modern Times]

SSDs often have a small very high performance cache, and that only gives performance benefits for a limited period.

Not likely to be the choice of OS, although some people do like to blame Windows for everything.

[Rebel]

You both make sense.

It might help to explicitly buffer large chunks of data, and write those out without any reading in between.

Such as the "copy" instruction probably does, copying files goes with 440 million bytes per second.

[Ras]

Such as the "copy" instruction probably does, copying files goes with 440 million bytes per second.

440MB/s sounds like an SATA SSD - an HDD probably wouldn't reach that, and the SATA interface bandwidth tops out at 600MB/s. An NVMe SSD should be a lot higher with sequential access. As for why, I think it's the SLC cache of the SSD. 2 bio positions with 40 bytes is around 80GB. Typical PCs have between 16 and 64GB of RAM, so that wouldn't even fit unless you have a PC with unusually much RAM. 80GB is on the order of maginitude I'd expect for the SLC cache of a 1TB drive, though that size often also depends on how full the drive is.

[chrisw]

I have this simple example code, copy file, 40 bytes per position.

Code: Select all

        fp1 = fopen(bin_one, "rb");
        fp2 = fopen(bin_two, "rb");

next:   x=fread(&k1,8,1,fp1); if (x<1) goto done;
        fread(&k2,8,1,fp1);
        fread(&k3,8,1,fp1);
        fread(&k4,8,1,fp1);
        fread(&k5,8,1,fp1);

        fwrite(&k1,8,1,fp2); 
        fwrite(&k2,8,1,fp2); 
        fwrite(&k3,8,1,fp2); 
        fwrite(&k4,8,1,fp2); 
        fwrite(&k5,8,1,fp2); 

        goto next;

done:   fclose(fp1);
        fclose(fp2);

This goes well the first 2 billion positions, average speed 6 million per second, then it starts to stall, average speed eventually goes down to less than 1 million per second.

Why is that?

Windows ?

Try C++ and streaming

[Rebel]

As we did in the past, even our C++ dataloader streamer started to slow down when NNUE data kept increasing, hence we don't use it any longer. I am fine with the solution to store my data into xxxx files of 10M positions and do a "copy /b" afterwards to make it one file for the trainer. The Windows copy command does not have this slow down problem.

What definitely helps is to trim SSD's before starting major disc operations.

[mar]

generally if you want to copy large chunks of data, you want to buffer a lot. reading/writing 8 bytes at a time isn't such a great idea
stdio probably has some small buffer by default, I don't think the copy command does anything magical other than using a big buffer
C++ streams wouldn't be any different than C stdio (in fact the API is horrible)

if we look at reactos source code for the copy command here: https://doxygen.reactos.org/d9/d15/base ... py_8c.html
all it does it allocates a page-aligned 16kb buffer and loops with ReadFile/WriteFile API, I also don't see any special hint flags when opening the files

[chrisw]

generally if you want to copy large chunks of data, you want to buffer a lot. reading/writing 8 bytes at a time isn't such a great idea
stdio probably has some small buffer by default, I don't think the copy command does anything magical other than using a big buffer
C++ streams wouldn't be any different than C stdio (in fact the API is horrible)

if we look at reactos source code for the copy command here: https://doxygen.reactos.org/d9/d15/base ... py_8c.html
all it does it allocates a page-aligned 16kb buffer and loops with ReadFile/WriteFile API, I also don't see any special hint flags when opening the files

Interrogate copilot a bit for C++ solutions and it eventually steps beyond streams and suggests using a different thread for read write and also some memmap code for Linux, for which it will also give a windows version if asked. Also states the obvious use a different SSD for read and write.
Tough for old bloke like me to work out the memmap code without trying to implement it first, and motivation lacking, but maybe way to go if you can save hours/days of copy time.

[JohnWoe]

C code and Windows. Memory leaking, throttling or Windows doing funny things. I would bet it will run flawlessly on Linux and C++ streams.