Limiting max number of Threads

I am using tigergraph in a shared server with a lot of cores and memory. Whenever I try to run a GSQL query tigergraph blocks all the available cores leaving the other users with no free cores.
Is there any way I can limit the maximum number of cores tigergraph can use when running GSQL queries?

Hi,

Welcome to TigerGraph community!

Well, normally TigerGraph runs on a dedicated platform/hardware and thus it is given all resources, as it is a database server, not just any application. It is actually a desired behaviour to use call CPUs as intensively as possible (for as short time as possible, i.e. do the work as quickly as possible). Production instances of TigerGraph should be run on dedicated instance(s).

If you need to restrict CPU usage, then probably you should run TigerGraph in a virtual machine or a container where you can specify the resource limits.

I will check it there is a way (some config parameter) to limit CPU/thread usage, but I would not be surprised if there was not (as it would contradict our intention to be the fastest graph database/platform).

1 Like

You can use cpulimit:
http://manpages.ubuntu.com/manpages/xenial/man1/cpulimit.1.html

Best,
Bruno

1 Like

Ok, thanks.
We are still testing if tigergraph is appropriate for our intended use, that’s why we have installed it in this server which is for general purpose. But of course, in production we would run it on a dedicated instance.

Thanks Bruno, will have a look at this and let you know if it works!

I checked all config parameters of 3.0.5 and there is none that would be applicable. Try @Bruno’s suggestion.

Thanks! I tried cpulimit and it worked.
However, I got another issue, because it took very long and it timed out (although I allowed it to use up to 100 cpus which I think was much more than enough for the task).

What timed out, exactly? I am asking this because we have query and REST endpoint timeouts that can be increased.

GSQL Client Session Timeout
RESTPP GSQL Query Timeout
GraphStudio Timeout (look at the end of the JSON config file)

I got this answer:
Failed to GET http://127.0.0.1:9000/query/aml3/subgraphAccAccDates?minDate=2019-01-16&maxDate=2019-02-15: HTTP 504 Gateway Time-out

I think it’s the REST Timeout maybe?
The query timeout (DefaultQueryTimeoutSec) I have already increased it to 36000 sec (10h).

What happen if you try to start this query using GraphStudio?

I need to use a vpn to connect to the server where tigergraph is stored and the vpn sometimes falls or kicks me out. So I prefer running things with nohup so that they will keep running even if my session stops.
I have tried other small queries (like getting the neighbors or 2 step neighbors of a node) with GraphStudio, which finished on time.
But this one has to search through all the nodes of a certain type and return only the ones between two dates. I supposed it would take long, but not this long :thinking:

Maybe I have to change one f these in the config GSQLConnectionTimeoutMS, RESTPPConnectionTimeoutMS?
They are both set to 120000.

@martaq you can try to set a query timeout by using GSQL-TIMEOUT:100000 option:
curl -X GET -H “GSQL-TIMEOUT:100000” “http://127.0.0.1:9000/query/aml3/subgraphAccAccDates?minDate=2019-01-16&maxDate=2019-02-15”

Bruno

I’m still getting this response:

> <html>^M
> <head><title>504 Gateway Time-out</title></head>^M
> <body>^M
> <center><h1>504 Gateway Time-out</h1></center>^M
> <hr><center>nginx/1.16.1</center>^M
> </body>^M
> </html>^M

How long the query is running until you get the 504 message?
Try to increase the GSQL timeout value to something above 600 sec.

The query runs for 2 hours and then it outputs that.
I set the timeout to higher than that:
curl -X GET -H “GSQL-TIMEOUT:360000000” " “http://127.0.0.1:9000/query/aml3/subgraphAccAccDates?minDate=2019-01-16&maxDate=2019-02-15

Maybe we should talk about improving the query :smiley: ?

Did you changed RESTPPConnectionTimeoutMS and GSQLConnectionTimeoutMS too?

No I didn’t change those, they’re still 120000.
Should I change them?

Yes please.
Btw did you found anything suspicious in the log files?
(log/nginx/logs/nginx.error.log , log/gsql/log.ERROR, log/gsql/log.INFO)?

Maybe you can start curl with -v (verbose) option, to see what is going on on the client side.

Ok, I tried again with different queries after setting the timeouts to a higher threshold and I still get the same answer and didn’t fnd anything suspicious in the log files, this is what I get adding -v option:

> GET /query/aml3/subgraphAccAccDates?minDate=2019-01-16&maxDate=2019-02-15 HTTP/1.1^M
> Host: localhost:9000^M
> Authorization: Basic ************=^M
> User-Agent: curl/7.65.3^M
> Accept: */*^M
> GSQL-TIMEOUT:360000000^M
< HTTP/1.1 504 Gateway Time-out^M
< Server: nginx/1.16.1^M
< Date: Wed, 11 Nov 2020 14:50:17 GMT^M
< Content-Type: text/html^M
< Content-Length: 167^M
< Connection: keep-alive^M
< X-Frame-Options: SAMEORIGIN^M
* Connection #0 to host timon left intact
<html>^M
<head><title>504 Gateway Time-out</title></head>^M
<body>^M
<center><h1>504 Gateway Time-out</h1></center>^M
<hr><center>nginx/1.16.1</center>^M
</body>^M
</html>^M

It’s interesting that the timeout is always after exactly 2hours, although, I don’t have any threshold in the config set to 2 hours…