Global warming--the Data, and serious debate

[grmorton]

No. The satellite data is quite different, and has very different systematics. The problem with the satellite data early-on was that they didn't take into account the slow orbital decay of the satellite, which led to both different times at which the data was read, and less atmosphere through which to read it. Overall this slowly, over time, biased the results to the cold side, and resulted in a temperature trend that was nearly flat. When they went back and corrected for this effect, the satellite data almost exactly matched the station data.

Yes, but I believe that the data I am using has been corrected for that. It is the Univ. Alabama Huntsville data. I will do some looking to try to verify that.

 

[thaumaturgy]

DETAILS:

In case Glenn is actually interested in "serious debate" about the "Data", I would recommend he read the following readme.txt file from NOAA about the "quality" and individual station reports:

ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/README.TXT

It is very enlightening as how the stations are assessed and kept track of.

This is hardly "sloppy", unconstrained "cowboy science" these folks are doing. I defy Glenn to take a look at the gigantic zip files exhaustively annotated (it's no easy matter because they are strings of digits and letters which must be de-encoded from the above readme.txt file to make sense of it, but indeed, they reveal the extensive quality control measures undertaken in regards to the USHCN surface station data.

 

[thaumaturgy]

Yes, but I believe that the data I am using has been corrected for that. It is the Univ. Alabama Huntsville data. I will do some looking to try to verify that.

So, praytell, Glenn why do you allow for orbital decay calculations but have a problem with the extensively detailed corrections clearly and plainly laid out by NOAA to adjust for data from surface stations?

Bit of "selective bias" there?

Why don't you cling to the raw unprocessed (or as you say "unsanitized" data)?

Seems only consistent.

(But then I'm not trying to defend your confirmation bias for you. Only you can do that).

Why don't you call the satellite data "crap" because it requires processing?

 

[thaumaturgy]

How to calibrate a pH meter.

In science we use many pieces of equipment day after day. It is necessary to, in most cases, calibrate that instrument daily. Such an instrument is a pH meter. pH meters can drift or the probe can become less effective. In my line of work silica and high pH tend to erode the probe rendering it less effective the more it is used. (* part of the problem of working with a glass probe.)

When I use a pH meter I can, if I so desire, use the raw electrical readout of current, but pretty much no one who uses a pH meter will do that. We do something that apparently Glenn skipped in his ISO9003 certification called calibration.

When I calibrate the pH meter I "teach" it every day what the electrical signal means in terms of actual hydrogen ion concentration.

This helps to take care of any drift in the instrument or any decay of the probe.

It is highly unlikely that if I just took the raw electrical signal from a pH 4 buffer at one day and tried to compare it to the same raw electrical signal of the same buffer several months later that the raw electrical signal would be the same, even if the hydrogen ion concentration was the same.

Now, of course, Glenn would, after one single use of a pH probe be required to throw it away (because he likes "unsanitized", "uncorrected" data). Any known drift which can be normalized is "cheatin'"

BUT if I calibrate the instrument (ie correct for drift or physical problems with the probe) I can get a reasonably good lifetime use of it. AND (I'm guessing Glenn hasn't used a pH probe since college?) the calibration usually gives me a slope by which I can assess how much longer I can keep using the probe.

At some point the calibration curve gets messy and I know the probe is trashed.

This is a simple example of what we who do science have to deal with on a daily basis.

Glenn wishes to use only "unsanitized" data (except in the case of the satellite data, apparently, but why he is so inconsistent is known only to him or his confirmation bias).

So when Glenn really wants to discuss data quality control I suspect this conversation will get a bit more interesting. (Of course it will get into more statistical detail, which is probably why Glenn has studiously avoided that except for a few weak comments on "standard deviation" as if that's all there is to stats).

So far Glenn has made a few freshman mistakes:

1. Assuming anecdotal data is somehow superior to statistical treatment of multiple lines of independent evidence treated with statistics

2. Assumed the data processing is somehow "cheatin'" (despite the fact that NOAA et al. have provided extensive explanation of exactly how data is processed to account for problems).

3. Failed to note that his accusations of some ill-defined "conspiracy" in processing of the Electra site data actually made a point directly opposite what he claimed (ie; the unprocessed data showed a leverage-induced temp increase, while the processed data showed no such increase)

4. Failed to address the f-test results on the original global temp data he presented.

Let this be an object lesson to those who wish to debate statistical data using anecdotal approaches devoid of any actual appreciation of how instrumental analyses are done.

(Glenn, is there any possibility I'm pressing every one of your buttons to the extent you are pressing mine? Because I sincerely hope that is the case.)

 

[grmorton]

You are an alarmingly priapic individual. But I expect nothing less of you.

I do admire the insult here (I really do. THat is creative, but the state of one's sexual organs is not per se the subject of this thread. Maybe you should try another thread.

So your thing is to look at unfiltered, unprocessed data. So any efforts to normalize for urban heat effects are removed.

Sigh, you clearly have forgotten the size of some of the towns we are talking about. We aren't talking about LA. Electra, for pete's sake has a population of about 3-4000. And you have clearly forgotten that quote I put of where James Hansen corrects the urban data by .3 deg C when NASA says the urban heat island effect is 2-4.5 deg C!

And, Let's be real here, isn't it amazing that urban heat island effect only happen on years when those darn volunteers and the MMTS's send in data that is 8-10 degrees more than the average temperature??? That is truly amazing.

Below is the corrections for Electra from 1952 to 1994. Isn't it amazing that no correction, or only tiny corrections were made prior to 1985. Then suddenly there was a need to do an 'urban' correction (for a town of about 3000) for the years 1985,1987, and 1987. But then suddenly in 1987, the heat island effect (if there really is one there) disappeared for 1988 and 1989, and returned again 1990, disappeared in 1991, reappeared in 1992 and disappeared in 1993 only to come back in 1994.

But the guy doing the editing on this data has god-like powers and can know that the exact correction to make for 1985 is 17.13 deg, yes, note that .13 degrees in the last significant digit. Such accuracy.

But wait, in 1986, this god-like editor decided that the correction was exactly 16.75 deg F and in '87 it was 16.51 deg F.

Since those darn volunteers got it right in '88 no correction was necessary. What utter silliness this is.

T, I can't hardly take you seriously when you think this is good data, or even correctable data with any degree of precision.

I am not going to respond to most of your posts because until you come to grips with the fact that the raw data can't be manipulated without introducing bias, or obtaining a god-like state in which one can ascertain what the true temperature was even though the data stream which is the observation, says it was something radically different.

At least I know now where you are getting your data from.

Do you do the same thing with the seismic data you recieve? Do you perhaps do any signal processing on it to make it make sense?

If so, how do you possibly justify such "shenanigans"?

We will edit the seismic data. But if we think a trace is bad, we simply delete it and don't use it. When there are noise bursts, we zero them out and don't try to know what we can't know. We don't put made up data into the seismic data. We zero it out and in our stacking process, or any process that requires averaging we only accept live values.

Many of those induced signals can be dealt with using perfectly reasonable assumptions, similar to how one processes signals in any data set. .

What possible method, other than ab fiat, do we decide that the correction is 16.51 deg in one year and only .03 in 1989??? Clearly they are judging that the temp in 1988 is clearly bad, but to say therefore that you can correct it seems hubris in th e highest.

Now, of course when you look at the raw data form you may be seeing issues from station movement or some known factor that caused the jump. To then draw a conclusion from that data you are further making an assumption that the scientists who use this data have made a similar faulty conclusion that you were drawing.

They don't even have records of all the station moves. Yes, they have some records of station moves, but the University of Arizona's parking lot thermometer, which can be seen on google earth, hasn't moved. The university has built up around it. They clearly had to move it long enough to put cement beneath it.

So you are concerned about seeing a huge jump in average temperature! Yikes! But if the people who keep track of this can in some way correct for the offset then it would appear from the ORNL mirror site data set that they have done this very thing.

But they can't correct for it except in an ad hoc way. This is probably the time to bring out a Geophysical REsearch Letters article that talks about the change from the RAW to the FILENET data, which is what you were using. Statistical analysis shows that the corrections are causing the warming!

“The annual difference between the RAW and FILNET
record (Figure 2) shows a nearly monotonic, and highly statistically
significant, increase of over 0.05 [deg]C /dec. Our analyses of
this difference are in complete agreement with Hansen et al. [2001]
and reveal that virtually all of this difference can be traced to the
adjustment for the time of observation bias. Hansen et al. [2001]
and Karl et al. [1986] note that there have been many changes in
the time of observation across the cooperative network, with a
general shift away from evening observations to morning observations.
The general shift to the morning over the past century may
be responsible for the nearly monotonic warming adjustment seen
in Figure 2. In a separate effort, Christy [2002] found that for
summer temperatures in northern Alabama, the correction for all
contaminants was to reduce the trend in the raw data since 1930,
rather than increasing it as determined by the USHCN adjustments
in Figure 2.It is noteworthy that while the various time series are
highly correlated, the adjustments to the RAW record result in a
significant warming signal in the record that approximates the
widely-publicized 0.50 C increase in global temperatures over the
past century.”

So again you are stuck with a strawman of sorts.

Not really. Statistics are on my side.

And I would poiint out that the raw data, with its less increase in temperature fits the satellite data better than does the adjusted data.

We certainly realize that the conterminous United States
represents only 1.54 percent of the Earth’s surface area, and analyses
of that areal unit may have limited interpretations for any global
temperature record. Nonetheless, we show clearly that adjustments
made to the USHCN produce highly significant warming trends at
various temporal scales. We find that the trends in the unadjusted
temperature records are not different from the trends of the independent
satellite-based lower-tropospheric temperature record or
from the trend of the balloon-based near-surface measurements.
Given that no substantial time of observation bias would be contained
in either the satellite-based or balloon-based measurements,
and given that the time of observation bias is the dominant adjustment
in the USHCN database, our results strongly suggest that the
present set of adjustments spuriously increase the long-term trend.”

I agree filtered data doesn't show the whole story but do be clear your preference for the unfiltered data without explanation of how the filtered data came from the unfiltered raw data provides just as little insight.

See above. BAlling and Idso think there is bias in the correction factors. And I absolutely agree.

No, you are wrong. You are completely and wholly wrong. I find that amazing for someone who deals with seismic and geophysical data where signal processing is the name of the game.

This from the guy who is new to time series analysis. Look Thaumaturgy, I have studied this area for quite a while and I have more data than I have put out. I can't put it all out at once--like that Balling and Idso article. The fact that I have all this data at my fingertips is because I have spent quite a while looking at this data from all around the world. ANyone who edits data knows that one's biases go into the editing. If you don't know that, you haven't edited much data. Editing demands that you know what ought to be there so that you can get rid of values that are spurious. You can only determine a value is spurious if you think you know what the true value is.

You simply want to see confusing and messy data. It helps you make specious claims.

Did NOAA, NASA, Hansen, University of East Anglia, or anyone else claim that the Electra data set shows global warming???? Did they?

LOL. It, along with Susanville with its large jump, and Waterville Washington with its large jum, and Walden New York with its large jumps and Hallotsville and Flatonia Texas with their spikes, and China with their rotten data, all contribut to the final product. Everywhere I look I see data like this. If it is everywhere I look, then how good can the final product be?

If they did, where did they claim it? In fact, the processed data shows no such thing.

So what you have done is deliberately avoid the proper data set in order to confirm your claim that the data is somehow "crap".

See Balling and Idso--most of the warming is due to the 'corrections'. I was waiting for this moment to bring out Balling and Idso.

But the important point to make here is that the data has been treated not to provide some false premise in support of global warming but rather to make sure the data is as "consistent" as is humanly achievable.

Let's repeat what Balling and Idso say:

“The annual difference between the RAW and FILNET
record (Figure 2) shows a nearly monotonic, and highly statistically
significant, increase of over 0.05 [deg]C /dec. . . ..It is noteworthy that while the various time series are
highly correlated, the adjustments to the RAW record result in a
significant warming signal in the record that approximates the
widely-publicized 0.50 C increase in global temperatures over the
past century.”

Otherwise there would be no way to ever produce a century's worth of useful data because the minute a station was moved or encroached upon by human civilization you'd have to cut off the measurement.

Maybe, but isn't that better than trying to claim that the globe is warming when it is only the cities and their urban heat island effect?

So you honestly think that this is cheating? If it is it's cheating in favor of the folks who don't believe in global warming.

See the quote from Balling and Idso above. They say it.

In point of fact the "cheating" as you call it is explained in a number of different places

Thaumaturgy, I am sure you are a nice guy, but you have no skepticism and you haven't dug deeply into these issues, not nearly as deeply as I have. In spite of your protestations, I know more about this than you, which is why you keep being on the losing end of these things.

I may be priapic as you say, and that is one of the more interesting things that has been said about or to me, but I do have one trait--go with the data, don't believe the hype.

No one is making data up. They are processing a noisy signal.

A fine statement of faith on your part. What was it that Balling and Idso said again?

“The annual difference between the RAW and FILNET
record (Figure 2) shows a nearly monotonic, and highly statistically
significant, increase of over 0.05 [deg]C /dec. . . ..It is noteworthy that while the various time series are
highly correlated, the adjustments to the RAW record result in a
significant warming signal in the record that approximates the
widely-publicized 0.50 C increase in global temperatures over the
past century.”

Again, if people believed in global warming simply because the Electra California RAW DATA SHOWED THE UPTICK then you'd have a valid criticism.

But Electra is only the poster child for the problem, as is Walden NY, etc. Let's see something else that Balling and Idso say

We compare the USHCN data with several surface and
upper-air datasets and show that the effects of the various USHCN
adjustments produce a significantly more positive, and likely
spurious, trend in the USHCN data.” Robert C. Balling and Craig D. Idso, “Analysis of adjustments to the United States Historical Climatology
Network (USHCN) temperature database, GEOPHYSICAL RESEARCH LETTERS, VOL. 29, NO. 10,, p.1387

The raw data is obviously available, and just like in all science, signal processing can and is applied every day to a number of different things.

In chemistry I subtract background spectra from FTIR's all the time. Am I cheating?

If you let your bias in, yes.

Sorry, but you are showing your confirmation bias. It is good to see the raw data but your "conspiracy theory" crap doesn't hold water. ESPECIALLY WHEN THE "SANITIZED" DATA MAKES YOUR POINT BETTER THAN YOUR RAW DATA DOES!






Quod Erat Demonstrandum!

QED not so fast. Deal with Ball and Idso, something you clearly were unaware of. Also, for correction, I don't think it is a conspiracy. I think it is group think. I think it is their bias entering the editing--the clever hans phenomenon. But I do think it is incompetence for the guys in climatology to be aware of these huge jumps and not try to fix the stations so that they don't have to do all these nonsensical corrections.

With that, I have other things to do this afternoon. Forgive me if I don't answer something in your other numerous posts. But, if you want, I will post the gradual rise in the difference between the raw and 'fixed' data sets. It is quite illuminating.

 

[grmorton]

So, praytell, Glenn why do you allow for orbital decay calculations but have a problem with the extensively detailed corrections clearly and plainly laid out by NOAA to adjust for data from surface stations?

Bit of "selective bias" there?

Why don't you cling to the raw unprocessed (or as you say "unsanitized" data)?

Seems only consistent.

(But then I'm not trying to defend your confirmation bias for you. Only you can do that).

Why don't you call the satellite data "crap" because it requires processing?

Well, because the 'corrections' get greater every year meaning that in order for global warming to continue, they must continue to make bigger and bigger corrections to the data. See the picture below. It is from Balling and Idso. I would ask the same question of you, are you interested in a serious debate, or only confirmation of global warming. It seems that you excuse any and all problems the data has. When making 16 deg corrections, it is hard to say with certainty that the temperature on average has changed only 1 degree.

Note below that the bias gets bigger and bigger each year.

There are 3 possible reasons for this. Instrumentation is becoming progressively worse as time goes on.

The observers are getting progressively drunker with each succeeding year

Bias is entering the editing process.

What is your choice?

Also attached is a histogram of the corrections made to Electra. It is hardly due to random noise.

By the way, I do want to re-answer your claim about conspiracy. You clearly have not lived in a country where you know that the information is controlled. In one country in which I lived, everytime CNN came on with a story about the country, the TV would go black. At first you would think something had happened, but, happily, when the critical account was over, the TV magically re-appeared, working properly.

I was involved in one event in this country which would have made huge news in the west. THere was not a single peep in the local papers or TV about it. The reporters willingly do the bidding of the party in power. I see some of the similar tendencies on the part of our press. So, with all due respect, you haven't actually seen the power to control information for a political agenda. I have. Call me crazy if you want, talk to me again after you have lived in one of these places. Otherwise you are babbling from inexperience.

 

[thaumaturgy]

But the guy doing the editing on this data has god-like powers and can know that the exact correction to make for 1985 is 17.13 deg, yes, note that .13 degrees in the last significant digit. Such accuracy.

Wow. It's almost like you completely forgot to actually read the systems by which the data is processed. You are building a strawman, Glenn.

Sorry to put it so badly but you make it sound as if there's some single guy who is just "making up" data to fit some preconceived notion.

Too bad you can't prove that caricature. Too bad I've provided NASA and NOAA's methodologies.

T, I can't hardly take you seriously

"can't hardly"? Did you mean "cain't hardly" there Jethro?

when you think this is good data, or even correctable data with any degree of precision.

You seem to be studiously avoiding the real discussion around the published methodologies employed by NASA and NOAA.

But if you want to continue to make a strawman I'll understand.

I am not going to respond to most of your posts because until you come to grips with the fact that the raw data can't be manipulated without introducing bias, or obtaining a god-like state in which one can ascertain what the true temperature was even though the data stream which is the observation, says it was something radically different.

Well, at least we know your confirmation bias is evident. Because you don't have to address me. You have to address NASA and NOAA.

So don't bother addressing me...bother addressing the people who deal with the data.

We will edit the seismic data.

Oh my. Talk about hypocrisy and inconsistency. Sorry to hear that. But again, your "confirmation bias" is showing. You obviously make some selection of what is bad or good traces. You must be "god like" in determining that data which you will "correct" and that which you will throw away.

You do realize that "throwing away data" on a gut feel is called "cheatin'".

But if we think a trace is bad, we simply delete it and don't use it.

Now if there was only a way to do better than "think" a data is bad or good. Oh, there is...it's called statistical process control.

You might want to try using that instead of just your gut, there, Mr. ISO9003.

When there are noise bursts, we zero them out

Oh my. "Zeroing out" noise sounds like manipulation. I am saddened that you are so alarmingly inconsistent in how you deal with data.

What possible method, other than ab fiat, do we decide that the correction is 16.51 deg in one year and only .03 in 1989???

Try reading the abundant information available from NASA and NOAA. They apparently aren't trying to hide anything from you.

They don't even have records of all the station moves.

Indeed the data goes back a century or more! I wonder how many data sets you use that have instrumental data going back over a century that require no correction of any sort.

Indeed this is why statistical analysis of the data as well as confirmation of neighboring station data and confirmation by alternate methods are very important.

But they can't correct for it except in an ad hoc way.

I suggest you read the links I posted.

See above. BAlling and Idso think there is bias in the correction factors. And I absolutely agree.

Very interesting article. Thanks for posting that. (My first publication was in Geophysical Research Letters.

This from the guy who is new to time series analysis.

At least I've been honest in the debate. But I am also learning. But the funny thing is You have yet to address the F-test data and the alarmingly significant p-value from the original data set that proves at 99.999% confidence that there was a trend in the data.

Funny that. You talk so big about how I'm new to time-series analysis but yet you don't seem man enough to actually deal with standard statistics (which I've had a bit more experience with).

Funny how my honesty is what you are somehow trying to twist against me. I like that.

Look Thaumaturgy, I have studied this area for quite a while and I have more data than I have put out.

Yet, ironically enough, you don't seem to have anything to say about the F-test data. You are making comments about my lack of experience with time series analysis which I pointed out initially, yet you are strangely silent on your own statistical background except to show us an unrelated paper you are co-author on.

Remember how I said my first publication was in Geophysical Research Letters? Well I was the technician for data the conclusions of which weren't really my area of expertise. I could run the machine and give the data, but I am not an oceanographer.

I'll be honest here, Glenn, you've done waaay too much blustery talk about what kind of statistics you can handle and so far that appears to be:

1. Listing an unrelated article that was stats-heavy that were co-author on (last co-author as I recall).

2. Talking about "standard deviation" and ignoring any detailed cases where the discussion has gotten off into F-tests, p-values, confidence intervals and statistical hypothesis testing.

So I'm left with no other conclusion to draw than to assume you don't understand the statistics.

You are more than happy to point up my errors (most of which I pointed out before you got around to it), so let me return the favor and say I no longer believe you have the stats background to handle that part of the discussion.

I'd love to be proven wrong, but you've had more than 20 pages of posts to prove me wrong, and so far I'm not impressed.

I can't put it all out at once

I like that point. It reminds me of what someone I used to respect once wrote:

...who refused my offer to discuss the existence of the geologic column by stating "It's on my short list of topics to pursue here. It's not up next, but perhaps before too long."(LINKY)

Funny how all things come back around isn't it?

Thaumaturgy, I am sure you are a nice guy, but you have no skepticism

There's a difference between uninformed ignorant skepticism for skepticisms sake and reading what is actually put out there.

In spite of your protestations, I know more about this than you, which is why you keep being on the losing end of these things.

QED not so fast. Deal with Ball and Idso, something you clearly were unaware of.

Indeed. I plan on reading this article. It sounds quite interesting.

I hope you will take the time to learn some statistics beyond just freshman level as well. Remember, both of us are in need of some education.

(I am also interested to note that you fail to addres the pivotal point that your raw data shows temperature increase while the processed data shows no such increase. Ironic you should pull that bit out to prove some positive bias point.)

But I do think it is incompetence for the guys in climatology to be aware of these huge jumps and not try to fix the stations so that they don't have to do all these nonsensical corrections.

Read my post about calibration. And also re-read your own posts about how you process data from seismic. Then prove to me one is incompetence and one is just good science.

 

[thaumaturgy]

By the way, I do want to re-answer your claim about conspiracy. You clearly have not lived in a country where you know that the information is controlled. In one country in which I lived, everytime CNN came on with a story about the country, the TV would go black. At first you would think something had happened, but, happily, when the critical account was over, the TV magically re-appeared, working properly.

Oh my. That sounds really bad.

(And in case you are trying to make point, you are sounding like you are verging on conspiracy theory again...sorry to point that out. I'm still talking about the U.S. data, because I don't have the chinese data. I have lived in the U.S. and I am not so fast to decree conspiracy when it goes against my own confirmation bias as some on this board are.)

I see some of the similar tendencies on the part of our press.

I bet you do! It's part of the whole "Mainstream media" giant bias conspiracy machine! The press is all powerful! And everyone is hiding data. Too bad there isn't something like a system of people to vette this power conglomerate.

We are in the thralls of the Conspiracy.

So, with all due respect, you haven't actually seen the power to control information for a political agenda. I have.

Well, I'm awfully glad we have insightful conspiracy theorists like you to keep us on the level! I'm so glad you exposed the NASA and NOAA and various international data processing facilities like such horrid police states as BRITAIN (University of East Anglia) for the "Global Warming Meme Machines" they most assuredly are!

Call me crazy if you want

I'll settle for just calling you a "crank". I've seen your type too many times before (albeit you are far more well-informed and educated than most cranks). Some libertarian-leaning guy who sees shadowy figures around the corner. Statistics is exactly the kind of thing you'd avoid like the plague because it sometimes shines a light on the dark corners. And that ruins the fun for you.

, talk to me again after you have lived in one of these places. Otherwise you are babbling from inexperience.

Well, I've only travelled to China and Europe, haven't lived there. So you got me there. If you wish to use that as some new angle of "attack" on me, please feel free to.

My 20 years of doing science in academia, as a U.S. Government Scientist, as an oceanographic research chem tech at a major university research center, two chemistry postdoctoral positions and nearly a decade in industrial research and development with a focus on statistical design of experiment have left me woefully ill prepared to deal with the Giant Conspiracy.

 

[thaumaturgy]

NIST Process Modelling Statistics Handbook.

Glenn won't respond to this (because it's statistics and so far he's pretty limited in his responses to anything statistical except to throw out a few "standard deviations".) So I'll post it for any surviving readers of this thread.

NIST is the National Institute of Standards and Testing. They are the ones who pretty much are the final arbiters of how things are measured here in the U.S. and a major world-class body. They have provided a handy handbook to show how STATISTICAL PROCESS CONTROL is done with regards to modelling data. (Note to Glenn: they are likely part of the big "conspiracy" around global warming data, so you don't have to listen to them...unless you do any science anywhere, but otherwise feel free to ignore them.)

It's HERE: http://www.itl.nist.gov/div898/handbook/pmd/pmd.htm

The current discussion is obviously quite germane to that point.

Here's an interesting point about the assumptions in statistical data systems:

The most basic assumption inherent to all statistical methods for process modeling is that the process to be described is actually a statistical process. This assumption seems so obvious that it is sometimes overlooked by analysts immersed in the details of a process or in a rush to uncover information of interest from an exciting new data set. However, in order to successfully model a process using statistical methods, it must include random variation. Random variation is what makes the process statistical rather than purely deterministic. (ibid)

Further:

The overall goal of all statistical procedures, including those designed for process modeling, is to enable valid conclusions to be drawn from noisy data. As a result, statistical procedures are designed to compare apparent effects found in a data set to the noise in the data in order to determine whether the effects are more likely to be caused by a repeatable underlying phenomenon of some sort or by fluctuations in the data that happened by chance. Thus the random variation in the process serves as a baseline for drawing conclusions about the nature of the deterministic part of the process. If there were no random noise in the process, then conclusions based on statistical methods would no longer make sense or be appropriate. (ibid)

(emphasis added)

And indeed as I have shown both NASA and NOAA have gone to extreme lengths to document their methodologies around vetting the data.


Now, just as an aside, I thought I'd look at an example of how NIST deals with data that show both a linear trend and a cyclic trend is described here: http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc4462.htm

The second example is for the monthly CO2 concentrations data set. As before, we start with the run sequence plot to check for stationarity.

The initial run sequence plot of the data indicates a rising trend. A visual inspection of this plot indicates that a simple linear fit should be sufficient to remove this upward trend.

This plot contains the residuals from a linear fit to the original data. After removing the linear trend, the run sequence plot indicates that the data have a constant location and variance, which implies stationarity.
However, the plot does show seasonality. We generate an autocorrelation plot to help determine the period followed by a seasonal subseries plot
(ibid)

(emphasis added)


I did that with the original global data set we were discussing earlier:

Then I fitted a linear trend to the data to make sure I'd have an equation to pull out of the data:

Then I removed the linear trend using the equation generated by this fit and this is what the data looked like:

et viola the linear trend removed we can see the cyclicity better.

In fact If you were to run the statistics of a linear least-squares regression on this data you will find it has a p-value of 0.9981 indicating that it is effectively a "flat line" (ie no trend).

QED.

I suspect Glenn will avoid this post as he did back on Page 20 when I originally posted this discussion.

Again, I'll assume it's because he is incapable of dealing with the statistics and statisical formalisms presented.

He will, if he so chooses, explain why the F-Test appears as it does. He will, in a word, have to explain the detailed statistics.

I don't expect much.

 

[thaumaturgy]

The vast scale of the Global Warming CONSPIRACY!

Revealed in the pages of SCIENCE magazine!

"Human activities ... are modifying the concentration of atmospheric constituents ... that absorb or scatter radiant energy. ... {M}ost of the observed warming over the last 50 years is likely to have been due to the increase in greenhouse gas concentrations".

IPCC is not alone in its conclusions. In recent years, all major scientific bodies in the United States whose members' expertise bears directly on the matter have issued similar statements. (SOURCE)

(Emphasis added)

Oh my gosh! ALL major scientific bodies in the U.S. whose members' expertise bears directly on climate matters have issued statements that indicate that most of the observed warming over the last 50 years is likely to have been due to the increas in greenhouse gas concentrations.

Thanks be unto God almighty that a lone oilman and a handful of "hardened skeptics" whose expertise decidedly doesn't bear directly on climate matters have found the massive errors, ignorance and conspiracy!

 

[grmorton]

NIST Process Modelling Statistics Handbook.

Glenn won't respond to this (because it's statistics and so far he's pretty limited in his responses to anything statistical except to throw out a few "standard deviations".) So I'll post it for any surviving readers of this thread.

No, we are not going to get into the F-test, continuous probabilty or chi-squared distributions. Nor will many on this list understand the null hypothesis. We are not going to go further into nerd grenades. We did enough of it on Fourier. That certainly will leave us as the only two readers, which may be your point. To drive people away from the list. I see you didn't respond to the correction data by Balling and Idos. It seems this is your methodology--never respond to anything against your point of view. Why, in your opinion do you think that the correction value is growing year by year as published in the Geophysical Research Letters?

Below is the picture again for your pleasure.

NIST is the National Institute of Standards and Testing. They are the ones who pretty much are the final arbiters of how things are measured here in the U.S. and a major world-class body. They have provided a handy handbook to show how STATISTICAL PROCESS CONTROL is done with regards to modelling data. (Note to Glenn: they are likely part of the big "conspiracy" around global warming data, so you don't have to listen to them...unless you do any science anywhere, but otherwise feel free to ignore them.)

Group think. Have you ever had a psychology course? Do you wonder why Hollywood is hopelessly leftist and Texas hopelessly conservative? Group think. People tend to fall into line with those around them. Challenging zeitgeist is tough. Do you know what an Ising model is? Group think is about the same thing as an Ising model, only applied to social groups.

It's HERE: http://www.itl.nist.gov/div898/handbook/pmd/pmd.htm

The current discussion is obviously quite germane to that point.

Here's an interesting point about the assumptions in statistical data systems:

Please explain why the corrections at Electra CA are not random? THere are too many big steps in the data. Indeed, this data looks like the data in the Eukaryotic gene orientation paper I wrote.

Unlike you, I don't have to prove anything and I have nothing to prove other than that the raw data is crap and manipulated via bias. So, please explain the graph below--which you seem to be avoiding.

It seems that everytime I post something attacking the validity of the methodology, you want to go off into a communication form which few here will understand. Please address the point of this graph. Each year the correction has to be bigger to keep global warming going.

What was it that Balling and Idso said? Oh yeah

“The annual difference between the RAW and FILNET
record (Figure 2) shows a nearly monotonic, and highly statistically
significant, increase of over 0.05 C dec-1. Our analyses of
this difference are in complete agreement with Hansen et al. [2001]
and reveal that virtually all of this difference can be traced to the
adjustment for the time of observation bias. Hansen et al. [2001]
and Karl et al. [1986] note that there have been many changes in
the time of observation across the cooperative network, with a
general shift away from evening observations to morning observations.
The general shift to the morning over the past century may
be responsible for the nearly monotonic warming adjustment seen
in Figure 2. In a separate effort, Christy [2002] found that for
summer temperatures in northern Alabama, the correction for all
contaminants was to reduce the trend in the raw data since 1930,
rather than increasing it as determined by the USHCN adjustments
in Figure 2.It is noteworthy that while the various time series are
highly correlated, the adjustments to the RAW record result in a
significant warming signal in the record that approximates the
widely-publicized 0.50 C increase in global temperatures over the
past century.”

You can ridicule anything about me that you want to, but you can't seem to explain this picture and so you rant on unimportant issues, post repeated notes on things not relevant and state your faith in NASA etc, even though I have shown you repeatedly contradictory data within that community. This is sad.

 

[grmorton]

Oh my. That sounds really bad.

(And in case you are trying to make point, you are sounding like you are verging on conspiracy theory again...sorry to point that out. I'm still talking about the U.S. data, because I don't have the chinese data. I have lived in the U.S. and I am not so fast to decree conspiracy when it goes against my own confirmation bias as some on this board are.)

T, Let me relate two examples that you will call 'conspiracy'. When I first traveled to that country in the early 90s, we were advised that our rooms would be bugged. One colleague was like you. He thought it was all paranoia. I told him over and over that we were not going to discuss business in the hotel room but we would go out and take a walk. He thought I was nuts. During that month we were fed a watery rice porridge every morning--really bad stuff. About half way through our time, my colleague made a comment one night in our room. "Damn I am so tired of xiao mi that I could just scream. I wish we could have eggs for breakfast just once!" The next morning, we had eggs for breakfast. I wish I had thought of doing what he did in exasperation. edited to add, that was the only day we had eggs

We talked our way into a city which wasn't originally on our schedule. We wanted to see that field. For the first time in our travels a guy from the ministry of foreign affairs showed up at all our meetings. When he smiled, his eyes didn't smile. I don't know how you do that, but he could. He realized I spoke a bit of the language. He took me outside and tried to see if I could read his language. I couldn't except for a few words. While I was looking at the sign, he stood behind me and said something in his language. At that time, I didn't understand enough then to know what he said, but I suspect he was trying to get me to react. That night, after dinner, I came back into my room and noticed that my suitcase wasn't like I had left it. But of course, one isn't sure about such things, until one looks and sees the cigarette butt in the ashtray. He was sending me a message--it wasn't my butt, I don't smoke.

Things are different now there, but the old hands will tell you of opening a door and seeing all the listeners listening to the conversations in the hotel rooms. Then everyone would laugh and they would shut the door.

Just for the pleasure of saying this, the last week I was there, a deputy foreign minister took me to a North Korean restaurant in Beijing staffed by lovely young ladies who were North Korean Military. It was an interesting experience. I ate dog.

Yes, you have lived in the US and have no experience globally. In the UK, you have to pay a tax to have a TV set. The government really does have a truck that goes down the street trying to detect homes with Telly's who haven't paid the tax. There is no bill of rights there, they can walk in and see your TV if they want to. One friend I worked with when I lived there had a father who was a Royal customs agent. He could pretty much do what he wanted. The EU laws have slowed a bit of that down now. But you, living in the US have lived a sheltered life and don't see these things. So, enjoy your small view of the governments of the world. I have been in 26 different countries and seen much.

Now, answer why the difference between the Corrected and the raw data grows each year. We can have our political debate another day. and you can check my sanity or lack thereof another time. Now is the time to discuss data, not to present statements of faith that all is well and no one would do anything wrong.

 

[Naraoia]

No, we are not going to get into the F-test, continuous probabilty or chi-squared distributions. Nor will many on this list understand the null hypothesis. We are not going to go further into nerd grenades. We did enough of it on Fourier. That certainly will leave us as the only two readers, which may be your point. To drive people away from the list. I see you didn't respond to the correction data by Balling and Idos. It seems this is your methodology--never respond to anything against your point of view. Why, in your opinion do you think that the correction value is growing year by year as published in the Geophysical Research Letters?

To be fair I think most people who are scared off by nerd grenades have left long ago.

I for one would like to see you address the statistics as much as I'd like to see Thaumaturgy address Balling & Idos.

(And I'm not quite sure what you were trying to do with standard deviations either, given that you are discussing trends in the mean... but then two half-hearted semesters don't usually make people expert statisticians.)

 

[grmorton]

The vast scale of the Global Warming CONSPIRACY!

Revealed in the pages of SCIENCE magazine!


(Emphasis added)

Oh my gosh! ALL major scientific bodies in the U.S. whose members' expertise bears directly on climate matters have issued statements that indicate that most of the observed warming over the last 50 years is likely to have been due to the increas in greenhouse gas concentrations.

Thanks be unto God almighty that a lone oilman and a handful of "hardened skeptics" whose expertise decidedly doesn't bear directly on climate matters have found the massive errors, ignorance and conspiracy!

Stating this is not dealing with the data. YOu have at last been reduced to ignoring the data set before you and saying we should all beleive what we have been fed.

Are you aware that at one time all geologists thought continental drift was idiotic? It was consensus science, and all the geological organizations condemned the very idea.

Are you aware that all medical societies once held that ulcers were the product of stress not due to a bacteria. One guy had to drink H. pylori in order to prove them wrong. Consensus ain't science, and science aint consensus.

Are you aware that at one time all physicists thought that the cosmological constant was zero? It was true because every textbook said so. Today we know that that consensus was wrong.

Are you aware that Barbara McClintock was ridiculed for beleiving in transposons? It couldn't be true because everyone knew such things didn't happen.

Are you aware that until recently everyone thought RNA played only a minor role in life transferring genetic information from the nucleus to the ribosome. Today we know of all sorts of involvement by RNA in the metabolism of life. But, there was a time when the old consensus held it not to be so.

Now, you tell me that everyone beleives; it is consensus. We knew that before we started this debate, so you are merely being redundant here. Of course everyone 'knows' that global warming is happening. But that doesn't mean the data supports it.

If you like holding to consensus views, that is your concern. I think it is a worse position to be in as a scientist to be just one of the cows in the herd rather than being willing to challenge things.

Maybe this will show the weakness of consensus

But consensus will not do as a defining characteristic of science, because precisely what the sociologists of knowledge and of culture have been showing is that all cognitive systems that are going concerns in any society are legitimised and maintained by consensus - from the ritual cults of traditional societies to the religions of our forefathers.”

Yes, witch doctors agree with the consensus of their culture. If this is your entire argument, it is incredibly weak. Does science stop merely because you like this consensus?

[FONT='Arial','sans-serif'][/FONT]

 

[Chalnoth]

NIST Process Modelling Statistics Handbook.

Just fyi, you're falling afoul of the NIST guidelines you're quoting there. The problem is a bit subtle, in that the NIST statement isn't entirely accurate: one doesn't need actual randomness to model a process as noise. One only needs to know that process' relevant statistical properties. So it's perfectly possible to model a completely deterministic process as random noise, if you know those statistical properties.

Whether or not there's a significant trend in the signal, after all, depends entirely upon what sort of noise properties you assume. It's easy to come up with a pure noise signal that will produce exactly this sort of trend if you merely assume that there are natural variations of that magnitude and on that length scale.

This, then, swings the discussion over to not whether or not it's warming (which is shown trivially by a linear regression), but instead what is causing it. After all, if it is just natural variation, then we would tend to expect that it'll just drop off. This is the crux of many of th global warming denier arguments. To respond to this argument, one needs to examine the causes of the current warming.

The best way to determine what is and is not a cause is to look at other, secondary effects of the various potential causes, effects that would look very different if other sources were the cause. For example, if the warming was caused by an increase in solar irradiance, then we would expect to see a relatively uniform warming, with differences from region to region based only upon relative albedo.

The atmosphere, for instance, would warm relatively uniformly. This has not been seen. Instead, what we see is a cooling of the stratosphere, which is an expected effect from greenhouse gases as more radiation from the Sun is reflected back to the Earth than before, reducing the amount of radiation the stratosphere receives. The observance of this effect, combined with the close match between the measurements of climate sensitivity to CO2, the CO2 increase in the atmosphere, and the actual warming seen, together provide very strong evidence for greenhouse gases (set of by CO2) being the primary cause.

 

[thaumaturgy]

No, we are not going to get into the F-test, continuous probabilty or chi-squared distributions. Nor will many on this list understand the null hypothesis.

Nice random list of some statistical terms! I'm particularly impressed that you brought up chi squared!

But the real reason we likely won't be getting into these things is because I suspect you've now tapped out your complete store of statistical knowledge.

Excellent.

At least I proved (yet again) I'm more man than you because I at least followed you down the path of the fourier transform when you brought it up.

Now you won't even talk about the standard statistics.

Too bad. But it lets me know you simply aren't up to the task. Too bad.

Do please tell me how great you are at this, but keep up the good work of avoiding showing us all your mad skills.

We are not going to go further into nerd grenades.

LOL!

We did enough of it on Fourier. That certainly will leave us as the only two readers, which may be your point.

Well, to be fair, because you have now thrown in the towel on statistics in an inherently statistical discussion, it really only leaves one on here who is serious about the discussion.

The other one is more like a freshman chem student trying to interpret noise and anecdotal data as meaningful signal.

To drive people away from the list. I see you didn't respond to the correction data by Balling and Idos.

Hey, I need to get to a library, dude. I said I'd look at it.

But you're sounding like one big ol' hypocrite yourself because "I see you didn't respond to..." THE STATISTICS.

But remember, at all points in the game: I've gone out more on the line than you have. I don't respect someone who demands more of me than they are willing to give themselves. You brought up FFT, I followed down your path. We are talking statistics here (even your friends Balling and Idso deal with stats), so I suggest you repay me in kind.

The discipline and drive to study things outside of my comfort zone comes from my experience going for and getting a PhD. (You can call me Dr.)

Please explain why the corrections at Electra CA are not random?

Corrections are not necessarily random functions!

Look again at my post on calibrating a pH probe. I measure the pH of silica dispersions a lot. (In case that is unfamiliar territory to you, these are usually quite alkaline and tend to corrode the glass envelope of the pH meter). Corrections later done (switching out an entire pH probe) will be a decidedly non-random process.

I don't know where you got the idea that corrections would have to be wholly random. You'll have to find a citation for that one.

Unlike you, I don't have to prove anything

Is that why you ad nauseam keep reminding us of:

1. How long you lived in China
2. How long you've been studying this data
3. How rich your friend is who works with math
4. Your unrelated statistics paper rather than doing any stats on the current data
5. How ignorant I am on FFT
6. How many FFT you run in a day

and I have nothing to prove other than that the raw data is crap and manipulated via bias.

Well, too bad you'll need actual statistics and an appreciation of the entire data set (including satellite, borehole, seasurface and ocean temps) to prove that point.

But don't let your little "freshman anecdotal data game" get sidetracked with how real scientists do data.

So, please explain the graph below--which you seem to be avoiding.

Like you're avoiding the statistics? Ha ha!

It seems that everytime I post something attacking the validity of the methodology, you want to go off into a communication form which few here will understand.

Actually, no, you are completely wrong. I like science and I like statistics and I was rather under the impression you were interested in a real discussion. I didn't realize when I started this you were unskilled in statistics beyond freshman level and you think that "anecdotal data" is somehow meaningful.

My bad. I will be unable to keep the conversation as simple and elementary school as you appear to want. It just won't happen, Mr. Morton. (Oh, and again, if you like you can call me Dr.)

 

[thaumaturgy]

Just fyi, you're falling afoul of the NIST guidelines you're quoting there. The problem is a bit subtle, in that the NIST statement isn't entirely accurate: one doesn't need actual randomness to model a process as noise.

Actually my reason for bringing up the NIST handbook was merely to show Glenn how noisy data is treated. Not specifically to any particular point.

Glenn is hung up on anecdotal data which means he is, by definition, just as likely to be interpretting noise as signal by focusing on noise.

This, then, swings the discussion over to not whether or not it's warming (which is shown trivially by a linear regression), but instead what is causing it.

Agreed. This is very important. Once we hammer out the data and how the data is treated, then the real work begins in showing the likelihood that it is anthropogenic.

The best way to determine what is and is not a cause is to look at other, secondary effects of the various potential causes, effects that would look very different if other sources were the cause. For example, if the warming was caused by an increase in solar irradiance, then we would expect to see a relatively uniform warming, with differences from region to region based only upon relative albedo.

Agreed. And the earlier discussed graphs of solar output from the Max Plank Institute data (since presumably the sun must be the source of that heat and if we can rule out an increase in its output during an increase in temp we've at least removed one potential factor).

 

[thaumaturgy]

Stating this is not dealing with the data.

You mean like you not dealing with the statistics?

Funny that. Hypocrite.

 

[thaumaturgy]

...

 

[thaumaturgy]

Hey Glenn, since I'm doing my homework on your Balling and Idso paper, do you care to do any homework on THIS paper?

An evaluation of the time of observation bias adjustment in the U.S. Historical Climatology Network

Vose, R.S.,[SIZE=+0] [/SIZE]Williams Jr., C.N.,[SIZE=+0] [/SIZE]Peterson, T.C., Karl, T.R.[SIZE=+0], [/SIZE]Easterling, D.R.

Geophysical Research Letters
Volume 30, Issue 20, 15 October 2003, Pages CLM 3-1 - CLM 3-4

The U.S. Historical Climatology Network (HCN) database contains statistical adjustments that-address historical changes in observation time at each observing station in the network. A paper in 2002 suggested that these adjustments cause HCN temperature trends to be "spuriously" warm relative to other datasets for the United States. To test this hypothesis, this paper evaluates the reliability of these "time of observation bias" adjustments in HCN. The results indicate that HCN station history information is reasonably complete and that the bias adjustment models have low residual errors. In short, the time of observation bias adjustments in HCN appear to be robust.(ibid)

Emphasis added.