There is a large amount of natural variation in the actual amount of crawling on any given day and ALSO variation in the accuracy of the data available for reporting in Webmaster Tools.

You need to observe the figures over a period of weeks/months and establish whethe the figures are normal fluctuations or trends showing that things are getting worse over time.

Note also that increased average download time can be due to various factors including:

--- the size of the pages and associated items (eg CSS and other files);
--- settings that you may or may not have updated;
--- the performance of your server (incl. whether dedicated/shared);
--- how busy your server was and any other sites on the same server platform;
--- service and routing issues at your hosting co. at any given time (permanent/temporary, scheduled/unschedulaed downtime, etc)
--- network connectivity and traffic level anywhere between google's computers and your servers;

For a typical site. your average page download time of 509ms is not particularly high, although I do understand that if you have been averaging 250ms and then doubled to 500ms it seems worrying at first sight.

Note also that google does not crawl all pages on the same day and give the average for all pages on the same day.   If some of your pages are much heavier than others, and googlebot happens to be crawling a group of "heavy" pages, the average nfigure reported will increase, even though nothing about your site's crawlability has changed.  

As an analogy, you might be amused to know that the reported average height of men and of women (based on "random" street sampling) varies depending on what time of day the survey is done!  Why?  Because you get a different range of folks on the street nduring the day compared with the late evening!!!

So basically, you should EXPECT to see fluctuations and variations; if the reported figures were always the same/similar, you should be suspicious about how they were produced.   However, if you see TREND then you need to investigate further.

Have you explored Google's experimental Labs Site Performance data (in your Tools account) and "Page Speed")?   Bear in mind that those are NOT crawl times - those figures are computed from data received from actual visitors via Google Toolbar.    Again, it is stressed they are just indicative not definitive.    Very useful, nevertheless.