Glu console holding up db connections

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Glu console holding up db connections

yvyas
We have seen this happen a couple of times in production where the glu console becomes unresponsive. Users are not able to log in and user's already logged in cannot even pick a fabric. The exception in the logs is :

2015/09/10 14:07:16.757 ERROR [JDBCExceptionReporter] [qtp2137918572-628] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].
2015/09/10 14:07:16.759 INFO [realm] Unable to authenticate with org.linkedin.glu.console.realms.ShiroLdapRealm - Hibernate operation: could not execute query; uncategorized SQLException for SQL [select this_.id as id7_0_, this_.version as version7_0_, this_.one_way_hash_password as one3_7_0_, this_.salt as salt7_0_, this_.username as username7_0_, this_.x509pem as x6_7_0_ from db_user_credentials this_ where this_.username=?]; SQL state [null]; error code [0]; [qtp2137918572-628] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [qtp2137918572-628] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000].

We talked to our DBAs and they mentioned that the max active connections is set to 1500 in production. How can we update the size of the jdbc pool, and what else can we do to prevent this from happening?

Let me know if I can provide more information. Thank you for your help!
Reply | Threaded
Open this post in threaded view
|

Re: Glu console holding up db connections

frenchyan
Administrator
Which version of glu are you using? There is this bug in older versions of grails https://jira.grails.org/browse/GRAILS-11043 which may affect old versions of glu. 

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Glu console holding up db connections

yvyas
We are running glu 5.6.1 in all our environments.
Reply | Threaded
Open this post in threaded view
|

Re: Glu console holding up db connections

frenchyan
Administrator
That's the latest version so I guess that should not be the issue.​ Would you say you have very high traffic/huge number of users banging on the database at the same time? It seems hard to imagine that all connections would be held for more than 30s at any point in time even with high volume.

You can probably increase the size of the pool, but if it is a leak it is very unlikely that it will resolve the issue. But this documentation page should help you out:


I believe the maxActive properties is the one you need

Yan
Reply | Threaded
Open this post in threaded view
|

Re: Glu console holding up db connections

frenchyan
Administrator
In reply to this post by yvyas
You can also check the "Note" section in the glu documentation about how to add properties in the Meta Model


Yan
Reply | Threaded
Open this post in threaded view
|

Re: Glu console holding up db connections

yvyas
In reply to this post by frenchyan
Hey Yan,
Yes we do have a lot of users attempting to use glu, even some automated bot accounts. I will increase the max active connections first and see how that affects the problem. Thanks for your input, appreciate it! I will let you know of the results.