After upgrading to Rails4, I started to see this error
ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query: in my logs. It seemed random in some places (normal, but generally larger than average queries), and less random in others (high failure rate in long running queries, e.g. in workers).
It turns out, I had inadvertently set the
reaping_frequency to 10s during my upgrade to Rails4. It wasn’t Rails4′s fault, but part of the config changes I made around this upgrade.
At some point in Rails, 10 seconds was flagged as the default (replacing the previous “no reaping” default). This caused problems, and was reverted. Sadly not before the practice spread over the internet, including into Heroku’s official recommendation for threaded servers which I implemented, and at the time of writing is still Heroku’s recommendation.
ActiveRecord’s connection reaper is designed to remove “dead” connections, and it seems this was promoted as a good thing to do at the time, especially in multi-threaded environments. However, it appears that it does not function correctly and can kill alive connections too. Rails issue #9907 indicated that it can kill longer running queries, and the Rails commit message when it was disabled by default (again) indicated it could cause segfaults in multi-threaded environments. Sounds bad all-round!
Searching my codebase for
reaping_frequency and removing all traces of this config (especially the
|| 10 on the config line recommended by Heroku – which was enabling the reaper, with a frequency of 10s by default) fixed the issue.
If you needed the reaper for it’s “good” side effects (as in you have connections that are for some reason actually dying that need clearing up), then I suggest it might be better to solve that problem another way, one with less side-effects, for starters you could try and stop them dying in the first place.
I recently launched a landing-page contest on 99designs that was unsuccessful (following 3 very successful experiences with 99designs, mostly for logos).
One thing I realised when looking at the responses to the contest, and re-evaluating my own requirements, is that I don’t actually want someone’s attempt at a landing page in Photoshop, but rather I want a landing page beautifully crafted in HTML5/CSS, and ready for me to use.
This is quite different to the “PS2HTML” service offered along side the contest where an image-based design is translated into HTML. The goal here is not to try and skip that step, and save money on the image-to-html translation, instead it is about designing with HTML/CSS in the first place. Frameworks such as Bootstrap have shown it is very possible to do html-first designs, and I believe that designing in this way will have a very different outcome on the final result compared to starting from a 2D image.
I suggested this while on the phone to Steven in support. He told me that in the distant past they did have an option for HTML for a bit extra, but what happened is that people tended to expect their entire website to be created even with a shopping cart system! I do understand how that situation would occur. Certainly I’m not suggesting a complete website service here. He also mentioned that many graphic designers are not very good at HTML – that I think is fine, this new category of contests would surely attract a new category of designers.
There would be some hurdles to overcome, such as to ensure people have the right expectations for such a contest (i.e. it won’t include a shopping cart, or PHP programming!). But I believe that the ability to launch a contest seeking a beautiful HTML5/CSS design would be a great addition to the existing image design contest option.
Failing that, here’s a startup idea for someone: “99designs for HTML” ;-)
I had a bad week this week trying to spin up some new Heroku dynos, getting loads of “pack-objects died of signal 13″ which is basically the connection being interrupted, even on our relatively small 130MB repo.
The Problem: the initial push to Heroku takes a while and can fail especially on large repositories. If you’re deploying an experimental branch for a new Heroku setup with a high probability of the deploy being refused, even if the push succeeds, it can be very time consuming and frustrating.
The solution? Use EC2 as a conduit.
That is, spin up an instance of EC2, checkout and deploy your code right from within the same datacenter that Heroku uses.
Rubinus is a ruby VM written in C with true parallelism support, and a cool design that attempts to write as much of the interpreter as possible in ruby itself.
It’s been developed for a few years now, with financial supported by Engine Yard and is stating to mature. Heroku have included it as pre-installed ruby option, so it’s now quite easy to use.
Here’s how I set it up in dev & on heroku:
I’ve been reading some very good things about Puma (all-Rack stack, proper thread support, & more). I gave it a try with the thread-friendly Rails4, on the thread-friendly Rubinius. To my surprise, the heroku+rails4+puma+rubinius performance was woeful. I’m not sure what I got wrong, but I was seeing 2 orders of magnitude slower requests than before, with most timing out. I’m not trying to bag Rubinius here, I think it’s a fantastic idea, but for whatever reason it isn’t working for me. Even though Heroku now supports it, perhaps it’s still early days – I’m going to leave it for a while and let it stabilise, at least to the 2.0 release.
So Rubinius didn’t work so great for me – but what about Puma itself? Puma suggests1 you use JRuby or Rubinius as they support true multi-threading, not the total lack of concurrency present in the RMI (the standard/reference ruby interprater). However, it turns out you can use it quite effectively as a Unicorn replacement simply by treating it like Unicorn.
In Unicorn you only have workers (forked processes) for parallelism. Puma introduces threads, and the default is 16. But it also has workers (which are not enabled by default). In a true concurrency environment you would normally have 1 worker per CPU core, and use threads for the rest – in the case of the RMI though, without operating-system-level threads, these workers are the only way to use OS concurrency.
So the key to make Puma behave like Unicorn is to specify multiple workers, but just 1 thread per worker. Your config will look like so:
# config/puma.rb workers 5 threads 1,1
That is, 5 workers (for my app, 5 is the golden number to work within Heroku’s 512MB memory limit) and 1 thread per worker. As long as you leave the threads at 1,1. Now your app is using Puma as a drop-in replacement for Unicorn, ready to be scaled with threads when your underlying Ruby implementation can support them.
Actually what I do on heroku is to use ENV variables (as they suggest for unicorn):
# config/puma.rb workers Integer(ENV["PUMA_WORKERS"] || 5) threads Integer(ENV["PUMA_THREADS_MIN"] || 1), Integer(ENV["PUMA_THREADS"] || 1)
That allows you to tune the performance with a simple
heroku config:set command rather than needing to push & redeploy.
To complete the setup, modify your Profile and Gemfile
# Procfile web: bundle exec puma -p $PORT -e $RACK_ENV -C config/puma.rb
# Gemfile gem "puma", "~> 2.0.1"
If you want to try it with rubinius, read my next post.
What about allowing just 2 threads per worker with Puma on RMI you ask? I ran a bunch of simple ‘ab’ profiles. It seems with any more than 1 thread, the threads just fall over each other and block horribly (not particularly surprisingly due to the GIL). It’s better to leave those requests in the Heroku request queue than jam them through a single worker. No doubt this will be a different story with a true-concurrency ruby engine, but until then my conclusion is that it has a distinctly negative effect.
I also tried doubling the workers on a 2x sized Heroku dyno. I didn’t get any performance benefit from this (it actually went down), so my conclusion is that two 1x dynos with 5 workers each is better than one 2x dyno with 10. As always, your milage will certainly vary!
Why bother with Puma if you’re going to treat it just like Unicorn? Aside from a small speed increase, and being thread-ready, Puma also handles incoming requests better. In Unicorn, the worker is tied up from the moment the client hits the server until the request is finished, so if the client request is slow, or the HTTP body is large, that worker is wasted. You can read more about that problem on heroku.
In conclusion: you can treat Puma like unicorn by setting the threads to 1, and using workers as you do in unicorn. The speed increase may not be massive, since you’re still process bound, but it’s a more modern server and ready to go multi-threaded when ruby is. Play around with your worker count, the perfect number is 1 below whatever number causes memory warnings on Heroku.
I was watching Ben Silbermann’s interview at D11 where he states (22:53):
The experience that I’m really interested is when someone new joins Pinterest on any device how do we make it incredibly easy for them to understand what Pinterest is and to pair them up with other people in the world that share their particular passions. I basically re-sign up for the service every day on different devices, and I think we have a long way to go and if we can do that, we can open the service up to people all over the world, we can expand the number of interests that are represented and so that’s the number one priority for the company.
This is a really good idea btw, I might take that one.
Despite this, I actually found Pinterest’s new user experience to be pretty bad. I like Pinterest, I think it’s a useful idea. When I signed up, it prompted me to follow a whole bunch of boards. Being new to the service, I figured I’d more or less just accept the default options and so I accepted all the suggestions and was soon following a whole bunch of boards (over a dozen).
Boards that for the most part were kind of annoying. Sure the occasional inspirational quote, or cupcake photo is nice. But I don’t need 1000 of them drowning out the things I have explicitly decided to follow.
And this is where I found the user experience very bad: unlike on Twitter where you can view all the people you follow on one page with 1-click per user unfollow them, I found no such way on Pinterest. It seems you have to click on the board name from the pin-stream, open in a new page then click on unfollow.
The other way you can find the boards you follow is from the menu, by selecting “Your Boards” and when that loads selecting “Following” (which a bit counter-intuitive, I don’t see how “Following” fits under “Your Boards”, since they are not yours). More counter-intuitively this is not actually a list of boards you follow, but a list of people from whom you follow at least 1 board. There is no unfollow button, and clicking on a user gives you a “Follow All” button, but no “Unfollow All” and the boards followed are not even highlighted. Even if you select a user, it will show their boards but does not highlight the ones you follow.
So I tediously unfollowed each board by selecting it from the pin-stream. Unfollow a few, reload the page, unfollow some more. [Aside: 1 board completely refuses to let me unfollow – every time I try it says "unfollowed" but it's pins stay in the pin stream nevertheless, but I digress I want to focus on the UX issue, not simple bugs, and at least it's just 1 out of 12+, that I can handle]. Eventually I got rid of all but that stubbon one. It would have been faster just to create a new account.
Since it’s so easy to follow boards, make it easy to unfollow as well! Give me a list of every board I follow (not every user whose boards I follow – that is useful only to follow more of their boards, and not for unfollowing), or at least highlight which boards of the user I do follow!, and allow me to one-click unfollow button. In other words, consider the 1-week user experience too :)
My issue is not the suggestions, that’s fine – just need a way to undo the suggestions. If the same thing happened to twitter, it would be trivial to clean up your account and drop the suggested follows in seconds.
A lot has been made in the Australian media about the cost of the NBN. The Liberal party wants to implement a cheaper solution – albeit one that is less future proofed, and may end up costing more when all is said and done (in the way a clunker is cheaper to buy initially). Actually, NBN Co isn’t actually spending taxpayer money on goods (like what happens when the country buys a fighter jet), but is instead borrowing money using Australia’s credit rating that it intends to pay back in full, with profit (typical of a business). But lets ignore the differentiation between buying goods and investing in money-generating businesses for a moment, and look at just the money being outlaid today. The NBN is projected to cost $37.4 billion1.
The government has raised $48.1 billion dollars to date from the sale of 89.1% of Telstra2. It still holds a 10.9% stake valued at $6.6 billion3. That’s $54.6 billion in total value to the government from the capital sale of Telstra.
From the money made from the sale of Telstra, the government can build one and a half NBNs. The good news is we only need one.
So let me ask: is it reasonable to invest two thirds of the money raised in selling last century’s telecommunications network to build this century’s? An investment, that is projected to pay itself back several times over, both through direct revenue and indirectly through taxation on economic growth that it will facilitate.
Plus, maybe we can sell it off in a hundred years and raise money for the next network. Why again do the Liberals want to build a FTTN clunker?
The gs-apng gem continues the good work of James Pozdena and Paul Gebheim with some key enhancements including automatic message truncation, correct UTF-8 handling, rails-independent JSON encoding, full 256 byte message support, more failsafes and logging.
As my brother succinctly put it “iOS likes to crash”. One such crash that can easily happen when you use background tasks is for your app to be killed for running the background task longer than the allotted time (generally 10 minutes).
The standard practice for avoiding this issue is to end the task you started with
beginBackgroundTaskWithExpirationHandler in the expiration handler block of the same call. That works well, but there’s one catch. The
UIBackgroundTaskIdentifier isn’t passed into that block, so you need to pass it through yourself. But what if some code modifies the variable you passed-in erroneously? If you are using a local variable, this isn’t a problem1 – but if your background task extends beyond one method it’s common to use a member variable, which could be accidentally modified.
And therein lies the danger. What if, despite all your very careful checking, that variable is modified. Now your failsafe won’t work correctly, and the app will crash. What’s worse, is it will crash in a totally random place. The crash logs give a very specific reason “App has active assertions beyond permitted time”, but don’t tell you exactly which LOC created the background task.
Here’s a better pattern. A safer failsafe if you will. Always declare the
UIBackgroundTaskIdentifier as a local variable. End the task in the expiration handler using that local variable. After the task is created, set your member variable to equal the local variable for any additional additional processing (such ending the task naturally when it completes). Now the expiration handler is guaranteed to end the correct task, regardless of what else happens.
This pattern generates a bit of boilerplate code, and lets face it the more you copy and paste the less maintainable things get, and the more chance there is for error. So I’ve wrapped up this pattern into a simple utility class. The method is
beginBackgroundTaskWithSafeExpirationHandler and it is designed to never crash with “active assertions beyond permitted time” no matter what you do. You can download the code from the gist.