After upgrading to Rails4, I started to see this error ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query: in my logs. It seemed random in some places (normal, but generally larger than average queries), and less random in others (high failure rate in long running queries, e.g. in workers).
It turns out, I had inadvertently set the reaping_frequency to 10s during my upgrade to Rails4. It wasn’t Rails4’s fault, but part of the config changes I made around this upgrade.
At some point in Rails, 10 seconds was flagged as the default (replacing the previous “no reaping” default). This caused problems, and was reverted. Sadly not before the practice spread over the internet, including into Heroku’s official recommendation for threaded servers which I implemented, and at the time of writing is still Heroku’s recommendation.
ActiveRecord’s connection reaper is designed to remove “dead” connections, and it seems this was promoted as a good thing to do at the time, especially in multi-threaded environments. However, it appears that it does not function correctly and can kill alive connections too. Rails issue #9907 indicated that it can kill longer running queries, and the Rails commit message when it was disabled by default (again) indicated it could cause segfaults in multi-threaded environments. Sounds bad all-round!
Searching my codebase for reaping_frequency and removing all traces of this config (especially the || 10 on the config line recommended by Heroku – which was enabling the reaper, with a frequency of 10s by default) fixed the issue.
If you needed the reaper for it’s “good” side effects (as in you have connections that are for some reason actually dying that need clearing up), then I suggest it might be better to solve that problem another way, one with less side-effects, for starters you could try and stop them dying in the first place.
I found a bunch of SO questions & bug reports on the mysql2 gem which could be caused by the reaper issue, and have suggested they try this fix. If others confirm that indeed their “Lost connection to MySQL server during query” issues on rails are solved by disabling this feature, I shall definitely petition Heroku to change their docs.
The possibly related SO questions are:
http://stackoverflow.com/a/22424215/72176
http://stackoverflow.com/a/22424181/72176
http://stackoverflow.com/q/15460959/72176
http://stackoverflow.com/a/22423893/72176
Possibly related issues on the mysql2 gem are:
https://github.com/brianmario/mysql2/issues/421
https://github.com/brianmario/mysql2/issues/495
Pingback: Mysql2::Error: Lost connection to MySQL server during query
I got here via the Heroku recommendation to set reaping_frequency in their recommended initializer for threaded web servers (as you mentioned, it’s still there). I had to investigate further as it was the first time I’d seen that configuration option. Thank you for taking the time to write up your article so that I now know to avoid it!
You’re welcome Abe. I have pointed this out to Heroku support, looks like they’re going to fix the documentation.
Think this issue really only applies to mysql + rails, so not sure that heroku (by default a postgres shop) needs to change its recommendations…
Heroku isn’t only a Postgres shop though, plenty use MySQL, when I started using Heroku they didn’t offer their own hosted database and MySQL was the recommendation.
Rails-core themselves reverted this change, my suggestion was that the docs mirror the rails-core best practice. If enabling this setting is beneficial for Postgres apps somehow (which is not proven, it seems like a bit of a hack to me), then perhaps it could be mentioned as an aside.