Recently I saw some strange behaviour in my NN code, so I tried to isolate the problem.
When I run the following program with the latest version of Visual Studio (v17.9.1) with both MSVC or the CLang compiler I get the following output:

#include <cstdint>
#include <limits>
#include <iostream>
using namespace std;
template <typename T> class bug
{
public:
void run()
{
T zero = T(0);
T min = numeric_limits<T>::max();
T max = numeric_limits<T>::min();
if (min > zero) min = zero;
if (max < zero) max = zero;
cout << min << " " << max << endl;
}
};
int main()
{
bug<int16_t>b1;
bug<int32_t>b2;
bug<float_t>b3;
bug<double_t>b4;
b1.run();
b2.run();
b3.run();
b4.run();
return 0;
}

I don't have GCC available, but I wonder if this problem also appears with GCC or whether it's just a Visual Studio bug.

For floating point types 'numeric_limits<T>::min()' does not return the largest negative value (like for integers) but the smallest value, this is what we call in Dutch 'een instinker'.

Joost Buijs wrote: ↑Sat Feb 24, 2024 10:09 am
I already found out what the problem is.

For floating point types 'numeric_limits<T>::min()' does not return the largest negative value (like for integers) but the smallest value, this is what we call in Dutch 'een instinker'.

That's really unintuitive. You need to use numeric_limits<t>::lowest. This will works for all types.