The actual problem is that for some beyond retarded reason in modules/update/update.manager.inc on line 706 (in my present setup, drupal 7.2) the following check is performed:
if (fileowner($project_real_location) == fileowner(conf_path()))
with the comments explaining that conf_path() resolves to e.g. sites/default (why the “e.g.” is beyond me - could it be something else?!? or is “i.e.” what the authors meant to say?).
The $project_real_location (how real could it be, it is a temporary location underneath temporary:/“) is where the file is downloaded and extracted.
Depending on the server setup, this can be randomly complicated by a number of factors. Fact is, even if the web server user has all the rights to write all over the place, it wouldn't make any difference. Enjoy wasting your time trying to make it do so.
And why select a location that is generally created manually prior to installation to check its ownership against files actually created by the web server user. There must be a better way to deal with this.
Eventhough, Wordpress is using a similar mechanism (someone is bound to have borrowed it in/from Drupal), they do have a configuration option that overrides this. This is in particular a problem when users can manipulate their setup, but cannot change ownership of files, and especially critical when performing backup, migration, or other maintenance tasks.
Even if there is no easy workaround for this, there has to be at least a hint clarifying exactly what the problem is and how to go about this in case it occurs.
Also note that the web server user may vary. On our setup it was assigned to games (id=5) for whatever reason, so intuitively changing the ownership to www-data (id=33) did not make any difference.
If there is absolutely no way around this, then include an empty file that serves this sole purpose - controlling the write access, but for goodness sake, do not rely on some odd side effects.
I could only join the above user(s) in their frustration! Way to waste people's time!
EDIT: P.S.: Please do excuse my tone, but I can imagine that the time spent on writing this particular piece of code is by no means comparable to the number of users affected by this and the amount of time they wasted trying to figure this out. The impact of this is quite large (just do a random search for issues related to this). Although it is not directly a security issue, it may have led many people to go against common security practices in desperate attempts to fix this (as some “solutions” appear to suggest).
Also, the amount of misinformation being spread regarding this issue, without understanding the actual problem, is quite astounding. There should be absolutely no need to fall back to FTP in this case or to manipulate permissions randomly in hope of fixing the problem (chances are it will not work anyway).
Just run this on your terminal (linux/ubuntu)
sudo chown www-data -R /var/www/html/websitepath/