Memory leak in Cfish ANN training

[Look]

Hi,

I am trying to train ANN for Cfish. After every thousand positions I seemingly have memory leak:

Code: Select all

[...]
    	if(cnt % 1000 == 0) 
        {       	
        	memset(&single_pos, 0, sizeof(single_pos));
        	free_all(pos);
        	init_gen(pos);
        	
        	if(cnt % 50000 == 0)
        	{
        	    printf("cnt is %lu\n", cnt);
        	}
        }
[...]

Called functions:

Code: Select all

void init_gen( Position *pos)
{
  	memset(pos, 0, sizeof(*pos));
  
  	pos->stackAllocation = SAFE_MALLOC(63 + 217 * sizeof(*pos->stack));

  	pos->stack = (Stack *)(((uintptr_t)pos->stackAllocation + 0x3f) & ~0x3f); 
  	pos->st = pos->stack + 7;
  
  	pos->moveList = SAFE_MALLOC(10000 * sizeof(*pos->moveList));
  	
  	pos->rootMoves = SAFE_MALLOC(1*sizeof(*pos->rootMoves));

  	pos->materialTable = SAFE_MALLOC(256*sizeof(*pos->materialTable)); 
  	
  	pos->pawnTable = SAFE_MALLOC(256*sizeof(*pos->pawnTable));
  
  	pos->counterMoves = SAFE_CALLOC(sizeof(CounterMoveStat), 1); 
  	pos->mainHistory = SAFE_CALLOC(sizeof(ButterflyHistory), 1);
  	
  	pos->captureHistory = SAFE_CALLOC(sizeof(CapturePieceToHistory), 1);
  	pos->lowPlyHistory = SAFE_CALLOC(sizeof(LowPlyHistory), 1);
  	pos->counterMoveHistory = SAFE_CALLOC(sizeof(CounterMoveHistoryStat), 1);
  
	return;
}

Code: Select all

void free_all(Position *pos)
{
	free(pos->st);
	free(pos->stack);
	
	free(pos->stackAllocation);
	
	free(pos->moveList);
	free(pos->rootMoves);
	free(pos->materialTable);
	free(pos->pawnTable);
	
	free(pos->counterMoves); 
	free(pos->mainHistory);
	free(pos->captureHistory);
	free(pos->lowPlyHistory);
	free(pos->counterMoveHistory);

	return;
}

Can you spot memory leak ?
How can I fix it ?

[Ras]

Code: Select all

  	pos->stack = (Stack *)(((uintptr_t)pos->stackAllocation + 0x3f) & ~0x3f); 
  	pos->st = pos->stack + 7;

Code: Select all

	free(pos->st);
	free(pos->stack);

pos-stack and pos-st are not allocated with SAFE_MALLOC, so they shouldn't be freed. Looks like undefined behaviour to me.

[Look]

Code: Select all

  	pos->stack = (Stack *)(((uintptr_t)pos->stackAllocation + 0x3f) & ~0x3f); 
  	pos->st = pos->stack + 7;

Code: Select all

	free(pos->st);
	free(pos->stack);

pos-stack and pos-st are not allocated with SAFE_MALLOC, so they shouldn't be freed. Looks like undefined behaviour to me.

I removed the extra frees . Still there seems to be memory leaks.

[Ras]

I removed the extra frees . Still there seems to be memory leaks.

Then they are in other parts of the code. For example, it could be that you accidentally null some of the pointers somewhere. Besides, what's even the point of that alloc/free cycle? You can simply memset the stuff to 0 if you need a value reset.

[Look]

I removed the extra frees . Still there seems to be memory leaks.

Then they are in other parts of the code. For example, it could be that you accidentally null some of the pointers somewhere. Besides, what's even the point of that alloc/free cycle? You can simply memset the stuff to 0 if you need a value reset.

I removed mentioned alloc/free code. Now in pos_set function , this is the crashing part according to debugger:

Code: Select all

void pos_set(Position *pos, char *fen, int isChess960)
{
  unsigned char col, row, token;
  Square sq = SQ_A8;

  Stack *st = pos->st;
  memset(pos, 0, offsetof(Position, moveList));
  pos->st = st;
  memset(st, 0, StateSize); // here
[...]
}

Where StateSize is:

Code: Select all

#define StateSize offsetof(Stack, pv)

I guess there is some missing allocation for pv in stacks of init_gen function.

[Ras]

Code: Select all

  memset(pos, 0, offsetof(Position, moveList));
  memset(st, 0, StateSize); // here

Without the data type declaration, that's impossible to tell, but it could be that these memsets are erroneously nulling some parts, including pointers, that they shouldn't be nulling. That would also explain why it didn't crash with the free, because you are allowed to free a null pointer, and why there's a memory leak.