I tried a better approach to the USHCN code. Previously I was averaging all monthly temperatures from all stations in a given year together. The problem with this is that some months have more station data than others, and is made worse by the fact that the USHCN adjusted data had extra manufactured temperatures in May. This caused the warmer temperatures in May to get weighted more heavily in the adjusted data than in the raw data.
The new approach is to average the months individually across all stations, and then take the average of all monthly averages in that year. This eliminates the issues caused by a larger number of stations with May data in the USHCN adjusted data set, than in the raw data set.
The result is a much smaller spike in 2014, similar to what Anthony plotted. But it doesn’t change the fact that using my approach the adjustments go up rapidly after 1995, whereas Anthony’s tails off flat. That discrepancy is the $64,000 question.
That’s better.
Thanks for getting me thinking about this. I am very intrigued now to understand the divergence after 1995 between your methods and mine.
Integrity. A rare commodity these days. Thanks and please keep up the good work. “Facts, not opinions.” David Kirkaldy
The difference is straighforward enough. Even if you use monthly rather than annual averages of absolute temperatures, you will still run into issues related to underlying climatologies when you are comparing, say, 650 raw stations to 1218 adjusted stations. You can get around this issue either by using anomalies OR by comparing the 650 raw stations to the adjusted values of those same 650 stations.
The reason why the 1218 to 650 comparison leads you astray is that NCDC’s infilling approach doesn’t just assign the 1218 stations a distance-weighted average of the reporting 650 stations; rather, it adds the distance-weighted average anomaly to the monthly climate normals for the missing stations. This means that when you compare the raw and adjusted stations, differences in elevation and other climatological factors between the 1218 stations and the 650 stations will swamp any effects of actual adjustments (e.g. those for station moves, instrument changes, etc.). It also gives you an inconsistant record for raw stations, as the changing composition of the station network will introduce large biases into your estimate of absolute raw station records over time. Using anomalies avoids this problem, of course.
In other words, they are fabricating data and generating fake temperatures – which very conveniently cool the past and warm the present, to turn an 80 year cooling trend into a warming trend. Got it.
Steve,
good to see that you can take on board comments adjust what you’re doing and refine your theory, not only that what you’ve been saying has been proved correct, and now it’s just a matter by how much has it been adjusted,when and who is the culprit True science, well done.
I think you may have (to mix metaphors) set the cat amongst the pigeons and set a hare running by highlighting these adjustments. It’s another weapon in our armoury when arguing that the climate is doing what it always has, and man’s influence is not measurable.
That is what science is about.
Climate alarmists think that science is some rigid entity of absolute truth. I see it is a detective story peeling back the layers of what is behind this scam, and how the scam works.
Steven Mosher finally long-format posts his mind:
http://wattsupwiththat.com/2014/05/10/open-thread-weekend-20/#comment-1633370
“Steven Mosher says:
May 10, 2014 at 8:48 pm
“Jimmy Haigh. says:
May 10, 2014 at 7:38 pm
One question I have is about TOBS. (Time of Observation). Why not just take the max and min temperatures regardless of what time they occur?”
Here is an experiment that EVERY SKEPTIC can do.
why? because it was already done and posted YEARS AGO on John Daly’s site
The experiement is simple. You take stations that record data hour by hour or minute by minute
You then Set your time of observation to be 7 am.
record the max and min.
Then Using the EXACT SAME DATASTREAM you change the time of observation to
1am, 2,am ect.
And you will see with your very own eyes how changing the TOB changes the min max
Historically in the US ( one of the FEW countries to change its TOB) the change results in cooling the past
Lets wind the clock back to 2007. 2007.
climate audit
http://climateaudit.org/2007/09/24/tobs/
here is the comment
http://climateaudit.org/2007/09/24/tobs/#comment-107763
Now, in 2007 I was highly skeptical of adjustments until I found JerryBs work
http://www.john-daly.com/tob/TOBSUM.HTM
“When using a pair of min/max thermometers for daily temperature observations, the time of day at which the readings for the previous 24 hours are observed, and the thermometers are reset, will often cause a time of observation bias (TOB). If readings are taken near the times of daily highs, or daily lows, those highs, and lows, often affect the readings of two days. Annual averages of the effects of TOB on recorded temperatures can be more than 1° F (0.56° C) at many locations, and near 2° F (1.11° C) at some. (This review of TOB is limited to temperature observations using min/max thermometer pairs, and/or electronic min/max thermometer sets which yield comparable results. Temperature observations using other kinds of thermometers may also have some kinds of TOB, but they are outside the scope of this review.)
In the “United States Historical Climatology Network” (USHCN), one kind of temperature adjustment is a TOB adjustment relative to midnight for observations made at times other than midnight. The occurence of TOB, and adjustments for it, are particularly important factors if the time of observation at a weather observation station changes.
In order to gain a better perspective of this bias, hourly temperature data of 190 US locations were used to calculate estimates of TOB relative to midnight, as well as estimates of some other items that seemed interesting.
The approach used is to choose several hypothetical “times of observation”, and to calculate what high, and low, temperatures a 24 hour min/max thermometer set would have “observed” at those times based on the hourly temperature records. These estimates cannot be precisely accurate partly because hourly observations will miss highs, or lows, that occur between the times of those observations, but hourly observations can provide at least an approximation of TOB. ”
here is my challenge. Ive made it many times on WUWT.
Go through the data. Do the work. I did back in in 2007.
to date I dont know of another person who has been willing to actually look at data on this.
when you do you will see that the TOBS adjustment is absolutely REQUIRED.
and ya, the US is one of the few countries that had to do this ( norway, canada, australia, and japan have a few isolated examples )”
I had a min/max thermometer when I was seven years old. It took about two days to figure out that I had to reset the thermometer at night in order to avoid double counting min or max temperatures. I’m sure that the vast majority of station owners figured this out too. TOBS is not a subtle problem. It is a blatantly obvious problem if you are recording temperatures with a min/max thermometer, and the solution for it is very simple.
That being said, the actual USHCN TOBS adjustment being done by USHCN doesn’t even vaguely match their documentation.
Steve,
Surely adding data for “missing” stations is also an adjustment?
However taking the time dimension down from one year to one month clearly removes weighting caused by data density which is not a relevant factor in this analysis.
I have just posted these comments and questions on WUWT regarding “Steven Mosher says:
May 10, 2014 at 8:48 pm on TOBS.
Steven Mosher says:
May 10, 2014 at 8:48 pm
I have just gone back to the data posted in the J Daly thread.
One aspect that has seems to have been totally ignored for Glass/Mercury Thermometers is when statements are made like “and 8 AM would all have indicated a low of -5.6 from the previous morning at 8 AM.”
How can the reading from the previous day be carried over in to the next 24 hour period when the Thermometer is supposed to be Reset with the magnet supplied each time the reading is taken.
ie if the reading was taken at -0.6 by 11 AM the previous day the “Indicator would be rest to -0.6 not left indicating -5.6 which has already been registered as the low for that day.
What do the proposers of TOBS think the Magnet was supplied for?
Can we have someone who has actually run an official weather station explain what the “Official Procedure” was for taking the temperatures?
I am with you on this one. Tobs would just shift the min or the max one day over. The thermometer is still getting the min and max for one 24 hour period no matter what ’24 hours’ you pick.
If you picked the hot of the afternoon as your Tobs you might get two very high max readings instead of one reading if you had a paticularly hot day or you might get two very low min readings instead of one reading if you picked dawn as your Tobs. However as Steve said most people would figure that out pretty quickly. Also the chances are the readings would be done in the evening after the regular work day was over. The second most common time to pick would be dawn or a bit after before the work day started.
I should add that over at WUWT a few years ago someone did post an old handbook 1800s? from the US army? on how the data was to be taken.
Sorry my memory is a bit fussy on the details and I did not bookmark the info. However the take home is there was a written procedure for the method.
Also Evan M. Jones/Anthony Watts are doing the corrections the correct way by going through the readings individually for Anthony’s paper on the Surface Station Project.
It will be interesting to see what happens when the data is treated with respect and the effort is put in to do it right and not to do it quick via an arbitrary Tobs ‘Adjustment’
Just as an aside to all of this recent argument:
The True Believers love it when Skeptics fight with each other. They can’t seem to grasp, though, that such infighting only serves to disprove many of their claims about skeptics.
The issue of methods was discussed (sort of) between yourself, Anthony and the other gentleman. An improvement has been achieved. Compare that to what happens when one of mikey mann’s buddies disses his work: snarling, rage, hissy fits…. or for that matter, any other climate scientist dares to point out something is not quite right with what they are doing.
Of course, that usually drives the dissenter further towards the Skeptic camps. And that is good for us. They might disagree, but Tol, Curry, Pielke Jr and a host of others have left the Religion of ‘man-made’ global warming, or at least, become heretics.
I’ve been keeping up on this over at WUWT and here…
http://wattsupwiththat.com/2014/05/10/spiking-temperatures-in-the-ushcn-an-artifact-of-late-data-reporting/#more-108904
Please follow the link back to Anthony’s site and scroll down to this gentleman’s comment:
James Hall (NM) says:
May 10, 2014 at 6:32 pm
Just a short thought regarding the Mercury-in-glass thermometers…
He speaks of real-world experience with temperature measurement. Informative.
What is the actual evidence, not a thought experiment, but actual evidence that people like James Hall were systematically committing TOB errors?
Steve
will you make your new code available?
Gail you are correct. All you have to do to prove or disprove a need to adjust the temps is to look at a simple sine wave, break it down into 24 sections depending on which section is your starting point you still have 24 sections the highs and lows will be at same points on the wave the only difference is, the starting time and ending time, over a 365 period (days) you will still have the same total of high points (365) and the same number of low points (365). Thus two stations next to each other will have sine wave that look different (because of the shift in recording times) but will be exactly the same. In order to get an exact reading with mean/medium/maximum/minimum you would have to record temps every few seconds to get within the tenths of degrees that the warming groups tries for.
People who do shift work (not the normal 9 to 5) will adjust their body temps (internally) to match the hours that is their “new” normal body rhythm. There is no need to add degrees of heat or cold to there recorded temps to make them in compliance with “normal”. Why would you do it to earths temps.
Skeptic are also more open to a discussion than the warmers as we are free from the dogma that is global warming. If you question any part of global warming you are labeled a heretic/skeptic. I prefer open debate and the skeptics side.
The data fraud has been proven by SG – you can quibble over how much – but the evidence is clear. All you need to do is to go back to any year 1990 to 1890 and look at what the posted temperatures were then and compare them to the ‘revised data’ from today. It is obvious to any but a door jam that the gov’t and their warming minions are cooking up a whole lot of fraud.