After denying the rumors that Google was taking site speed into account in rankings, it has now (April 9th) been officially announced that they ARE in fact considering site speed.
They also clarify that less than 1% of search queries are affected by this. So you can safely assume unless your site is REALLY slow, you don’t have to worry about it.
Here’s the full announcement including some ways to test your site speed.