Scorpio 2.8.7 MCTS+NN windows version

[Daniel Shawul]

Did you install tensorflow_cc to /usr/local after you compile "sudo make install" ?
It should put the include and lib files in there and you should have no problems.

Once you do that change USE_TF = 1 in the egbbdll Makefile. Right now it is defaulting
to the second approach USE_TF = 2 -- which is probably why you are getting those
errors with .pb.h files.

Btw, you don't need to download egbb files to use neural networks.

Daniel

[Joerg Oster]

Did you install tensorflow_cc to /usr/local after you compile "sudo make install" ?
It should put the include and lib files in there and you should have no problems.

Once you do that change USE_TF = 1 in the egbbdll Makefile. Right now it is defaulting
to the second approach USE_TF = 2 -- which is probably why you are getting those
errors with .pb.h files.

Btw, you don't need to download egbb files to use neural networks.

Daniel

Thank you.
Nice to know no egbb files are needed.

Although I'm pretty sure I did both, "sudo make install" after compiling
and changing USE_TF = 1, I will give it another try tomorrow.

[Joerg Oster]

Did you install tensorflow_cc to /usr/local after you compile "sudo make install" ?
It should put the include and lib files in there and you should have no problems.

Once you do that change USE_TF = 1 in the egbbdll Makefile. Right now it is defaulting
to the second approach USE_TF = 2 -- which is probably why you are getting those
errors with .pb.h files.

Btw, you don't need to download egbb files to use neural networks.

Daniel

Thank you.
Nice to know no egbb files are needed.

Although I'm pretty sure I did both, "sudo make install" after compiling
and changing USE_TF = 1, I will give it another try tomorrow.

The second try yields the exact same result and same problems, unfortunately.
Interestingly, the small example (inside the folder 'example') builds without any problems and outputs "Session successfully created."

I found two issues on the github repository, though, which seem to be closely related to my problem.
https://github.com/tensorflow/tensorflow/issues/1890 and
https://github.com/FloopCZ/tensorflow_cc/issues/121

[Daniel Shawul]

Did you install tensorflow_cc to /usr/local after you compile "sudo make install" ?
It should put the include and lib files in there and you should have no problems.

Once you do that change USE_TF = 1 in the egbbdll Makefile. Right now it is defaulting
to the second approach USE_TF = 2 -- which is probably why you are getting those
errors with .pb.h files.


Btw, you don't need to download egbb files to use neural networks.

Daniel

Thank you.
Nice to know no egbb files are needed.

Although I'm pretty sure I did both, "sudo make install" after compiling
and changing USE_TF = 1, I will give it another try tomorrow.

The second try yields the exact same result and same problems, unfortunately.
Interestingly, the small example (inside the folder 'example') builds without any problems and outputs "Session successfully created."

I found two issues on the github repository, though, which seem to be closely related to my problem.
https://github.com/tensorflow/tensorflow/issues/1890 and
https://github.com/FloopCZ/tensorflow_cc/issues/121

Did you build the shared or static library of tensorflow_cc? egbbdll needs the latter and you should have
libtensorflow_cc and libprotobuf.a in /usr/local/lib/tensorflow

For the shared library, you need to do
cmake -DTENSORFLOW_STATIC=OFF -DTENSORFLOW_SHARED=ON ..

What version of bazel are you using? At some point only 0.11 used to work for me but since you
already built it i guess this is no problem for you. The include directories i have in egbbdll Makefile should
be enough to build.

Also you might want to build and install both the shared and static libraries. I am not sure but initially i may
had similar issues as yours when i only had shared library installed...probably an issue with tensorflow_cc

Sorry i couldn't be of more help.

Daniel

[Joerg Oster]

Did you install tensorflow_cc to /usr/local after you compile "sudo make install" ?
It should put the include and lib files in there and you should have no problems.

Once you do that change USE_TF = 1 in the egbbdll Makefile. Right now it is defaulting
to the second approach USE_TF = 2 -- which is probably why you are getting those
errors with .pb.h files.


Btw, you don't need to download egbb files to use neural networks.

Daniel

Thank you.
Nice to know no egbb files are needed.

Although I'm pretty sure I did both, "sudo make install" after compiling
and changing USE_TF = 1, I will give it another try tomorrow.

The second try yields the exact same result and same problems, unfortunately.
Interestingly, the small example (inside the folder 'example') builds without any problems and outputs "Session successfully created."

I found two issues on the github repository, though, which seem to be closely related to my problem.
https://github.com/tensorflow/tensorflow/issues/1890 and
https://github.com/FloopCZ/tensorflow_cc/issues/121

Did you build the shared or static library of tensorflow_cc? egbbdll needs the latter and you should have
libtensorflow_cc and libprotobuf.a in /usr/local/lib/tensorflow

For the shared library, you need to do
cmake -DTENSORFLOW_STATIC=OFF -DTENSORFLOW_SHARED=ON ..

What version of bazel are you using? At some point only 0.11 used to work for me but since you
already built it i guess this is no problem for you. The include directories i have in egbbdll Makefile should
be enough to build.

Also you might want to build and install both the shared and static libraries. I am not sure but initially i may
had similar issues as yours when i only had shared library installed...probably an issue with tensorflow_cc

Sorry i couldn't be of more help.

Daniel

Your help is highly appreciated, of course.

I just successfully built the static library on my main PC.
At least, it didn't break with any error. Just the usual warnings.
As described on the tensorflow_cc github page, I strictly followed the commands to build the static version:

mkdir build
cd build
cmake ..
make
sudo make install

I have 3 files inside /usr/local/lib/tensorflow_cc: libprotobuf.a libtensorflow-core.a nsync.a

Unfortunately, I get the exact same error(s) when trying to compile egbbdll like on my other system.

Code: Select all

In file included from /usr/local/include/tensorflow/tensorflow/core/lib/core/errors.h:21:0,
                 from /usr/local/include/tensorflow/tensorflow/core/platform/env.h:24,
                 from eval_nn.cpp:4:
/usr/local/include/tensorflow/tensorflow/core/lib/core/status.h:23:10: fatal error: tensorflow/core/lib/core/error_codes.pb.h: Datei oder Verzeichnis nicht gefunden
 #include "tensorflow/core/lib/core/error_codes.pb.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Those *.pb.h header files are simply missing.

Question: If you were able to successfully compile egbbdll on your Linux system, shouldn't this also run on my box?
Maybe this is worth a try?

[Daniel Shawul]

Yes, I can upload the linux-cpu version.

Egbbdll does not currently build with the static version of tensorflow_cc. It used to but I removed the feature a month ago
because I wanted to have a separate tensorflow so file (though i have the opposite for windows version now).
The only reason I asked you to try and build the static library is because i thought maybe it will put the missing headers in
/usr/local/include/tensorflow. I remember i needed to build both initially for some reason ... So build the shared tensorflow_cc on top of the
static one if you haven't already done so.

I will upload the egbbdll and libtensorflow_cc later for you to try if that doesn't work.

Daniel

[Joerg Oster]

Yes, I can upload the linux-cpu version.

Egbbdll does not currently build with the static version of tensorflow_cc. It used to but I removed the feature a month ago
because I wanted to have a separate tensorflow so file (though i have the opposite for windows version now).
The only reason I asked you to try and build the static library is because i thought maybe it will put the missing headers in
/usr/local/include/tensorflow. I remember i needed to build both initially for some reason ... So build the shared tensorflow_cc on top of the
static one if you haven't already done so.

I will upload the egbbdll and libtensorflow_cc later for you to try if that doesn't work.

Daniel

OK, building the shared version atm.
Will report later if that worked.

[Joerg Oster]

Finally success!
After building the shared library, which took quite some time,
compiling egbbdll worked (with USE_TF = 1).

Only thing I needed to change was line 56 of the egbbdll makefile into

Code: Select all

TF_LIB += $(TF_DIR_LIB)/libtensorflow-core.a 

Now I hope the fun part can start!
Thanks for your help!

[Daniel Shawul]

Finally success!
After building the shared library, which took quite some time,
compiling egbbdll worked (with USE_TF = 1).

Only thing I needed to change was line 56 of the egbbdll makefile into

Code: Select all

TF_LIB += $(TF_DIR_LIB)/libtensorflow-core.a 

Now I hope the fun part can start!
Thanks for your help!

Ok nice ! It seems you are using the static library though. Here is the commit as to when I removed it -- but maybe it is time to bring it back.

https://github.com/dshawul/egbbdll/comm ... f6fd6ed5e6

The shared library built with --config=monolithic is libtensorflow_cc.so and this is the one that can be used with GPUs.

Also there is an inherent problem with protobuf and tensorflow shared library built outside of bazel.
I was not able to use multiple GPUs with tensorflow_cc and other prebuilt windows libraries as well -- and I spent
literally weeks chasing for a workaround. I couldn't find a solution so i resorted to building using bazel.
Your egbbdll binary will crash because of that where it tries to set the CPU/GPU number, so you need to comment out the following lines
in eval_nn.cpp

Code: Select all

    //std::string dev_name = ((dev_type == GPU) ? "/gpu:" : "/cpu:") + std::to_string(dev_id);
    //graph::SetDefaultDevice(dev_name, &graph_def);

regards,
Daniel

[Joerg Oster]

Finally success!
After building the shared library, which took quite some time,
compiling egbbdll worked (with USE_TF = 1).

Only thing I needed to change was line 56 of the egbbdll makefile into

Code: Select all

TF_LIB += $(TF_DIR_LIB)/libtensorflow-core.a 

Now I hope the fun part can start!
Thanks for your help!

Ok nice ! It seems you are using the static library though. Here is the commit as to when I removed it -- but maybe it is time to bring it back.

https://github.com/dshawul/egbbdll/comm ... f6fd6ed5e6

The shared library built with --config=monolithic is libtensorflow_cc.so and this is the one that can be used with GPUs.

Also there is an inherent problem with protobuf and tensorflow shared library built outside of bazel.
I was not able to use multiple GPUs with tensorflow_cc and other prebuilt windows libraries as well -- and I spent
literally weeks chasing for a workaround. I couldn't find a solution so i resorted to building using bazel.
Your egbbdll binary will crash because of that where it tries to set the CPU/GPU number, so you need to comment out the following lines
in eval_nn.cpp

Code: Select all

    //std::string dev_name = ((dev_type == GPU) ? "/gpu:" : "/cpu:") + std::to_string(dev_id);
    //graph::SetDefaultDevice(dev_name, &graph_def);

regards,
Daniel

Unfortunately, the static build of egbbdll doesn't load.
Even if I outcomment the mentioned lines in eval_nn.cpp.

Trying to build with USE_TF = 2 and the makefile (not with Bazel!),
I get an error.

Looking into the makefile, something seems strange:
ifeq ($(USE_TF),1)
TF_DIR=/usr/local
TF_DIR_INC=$(TF_DIR)/include/tensorflow
TF_DIR_LIB=$(TF_DIR)/lib/tensorflow_cc
else ifeq ($(USE_TF),2)
TF_DIR_INC=/home/dabdi/stests/vals/tensorflow <===
TF_DIR_LIB=$(TF_DIR_INC)/bazel-bin/tensorflow
endif

I guess I need to modify TF_DIR_INC to fit my system?