Global warming--the Data, and serious debate

[Chalnoth]

Just as a matter of interest here's some cyclical data with a clear secular trend:

and here's its spectral analysis:

So, impress me, MrMorton. Show me what that big peak on the left-hand side means?
Or more importantly show me how it cannot mean a secular trend.

Well, I think first you should remove the zero mode from the plot, because that mode means nothing. It's the ramp at the small-but-non-zero that is trend-like behavior. Here it looks so sharp that it's not clear by eye whether what we're seeing is mostly the zero mode or the small-but-nonzero ones.

 

[grmorton]

Big mea culpa in order:

Now I noted that, indeed the Leroy (1998) 1-5 ranking scale includes only positive biases. However clearly negative biases are possible.

However:

Davey and Pielke suggested that poor siting–induced bias could be positive or negative, (SOURCE)
​

The results presented here clearly support the theory that, if poor siting causes a bias, homogeneity adjustments account for the biases and contradict the hypothesis that poor current siting causes a warm bias or even any bias in the homogeneity-adjusted U.S. temperature change record. (ibid)​

​

Emphasis added.​

Here is the problem. A homogeneity adjustment is a change to the trend of the temperature stream. Since it is the temperture trend that we are trying to determine in the first place, taking data and forcing it to fit some pre-conceived trend, or even some arbitrarily determined standard, means that the data no longer speaks for itself. We take the data and tilt it to fit what we pre-conceive. And given that the temperature graph you showed, with a .6 deg change in the past 30 years, while the satellite data doesn't agree, one must ask if the homogeneity adjustment is the cause of the disparity??? It is either that or air conditioning.

Thanks for finally noticing that heating doesn't cause cooling. I will tell of another debate that I was involved in discussing energy generation and I made the stupid comment that there wasn't enough energy in some of the floating-in-the-ocean energy island schemes to power a ship. Then someone pointed out to me that sailing vessels proved me wrong. I had no choice but to blush! We have all been there. But thank you.

 

[grmorton]

Actually I don't.

I mistakenly assumed the LeRoy metrics on siting were for both positive and negative biases. I am obviously mistaken since they only site location near heat sources. However I will also note that biases can be both positive and negative in sitings as pointed out in the Peterson paper.

But you mistakenly think that tilting the trend fixes the problem.

Nice. Now's your chance to stop insulting me and "teach" a bit.

A few posts back, after you sneered, 'asking if your explanation went over my head', I said, "don't blame me for re-introducing the hostility" Do you recall that? I am sorry, but doing what I did seemed to be the only way to get you to see that your position was utterly untenable. In other words, you weren't listening when I had pointed out that being on cement, on a rooftop or next to an airconditioner was a bias. You specifically denied it over and over. I finally got your attention.

So, maybe it is time for you to stop being so [bless and do not curse][bless and do not curse][bless and do not curse][bless and do not curse] sure of yourself on issues you are new to and be willing to actually listen and learn rather than constantly assuming that anyone who doubts GW must be a nutter. How about that. Then I will try to teach a bit.

So your friend didn't think that a peak near zero on a periodogram could be an exceptionally long period trend? Since FT treats all trends as cyclical, what do you think the wavelength of something at that end of the spectrum is?

Because a peak at zero frequency is a constant shift. Don't confuse a ramp in frequency domain with a ramp in the time domain.

Now please explain to me how you would identify a secular trend in periodic data?

If you can, that is.

You would have to look at the phase diagram of the Fourier transform, and I will tell you, I know no one who can look at them and mentally interpret a phase diagram. The phase data jumps all over the place from one frequency sample to the other. A rising secular trend should have a phase in the low frequency range near zero. But the problem is that 2 cycles/sec may have it but 2.5 might not, then 3 will by 45 deg and 4 will be -87 deg.

Did you show your "laughing friend" the data?

Yes, I showed him exactly what we were t alking about. I apologize for telling you that, but the problem is, when one is so [bless and do not curse][bless and do not curse][bless and do not curse][bless and do not curse] sure that they are right, and they obviously think that the person they are debating with is so stupid that they no longer actually listen or respect what they say, sometimes the only way to break through is a bit of ridicule.

Just for an outsider's view, let's look at some temperature from Europe (SOURCE= ANALYSIS OF SECULAR TIME SERIES OF CLIMATOLOGICAL CHARACTERISTICS RNDr. Ladislav METELKA
Department of Meteorology and Environmental Protection
Faculty of Mathematics and Physics
Charles University
Prague

September 1997​

(LINK)

Just as a matter of interest here's some cyclical data with a clear secular trend:

and here's its spectral analysis:

So, impress me, MrMorton. Show me what that big peak on the left-hand side means?
Or more importantly show me how it cannot mean a secular trend.

Do you know what zero frequency is? If you thought about it you would know that a peak at zero frequency is a DC shift. It is a straight line parallel to the X-axis. It isn't a secular trend. The biggest number on the chart above is the DC shift and DC shifts are not secular trends. they are perfectly flat!

Or better yet have your "laughing friend" explain how he does time-series analyses such that secular trends cannot be modeled as just extremely long period cycles.

(Honestly, I'm curious because this all seems so common sense that when you find people who laugh at it, I have to learn the math behind it).[/qutoe]

It is a low frequency phenomenon as I said, but the phase plot is essential. And as I said, I have never met anyone who actually intuitively understands the phase spectrum. They always look like scatter-grams.

Oh and one other thing, in case you are interested in misrepresenting what I've asked or said here, I will remind you that some time series analyses treat secular trends as nothing more than just a very long period cycle, I suppose the key then is to limit drawing conclusions on a wavelength well outside of the sampling window and how you differentiate that from a pure secular trend.

But the big peak on the side is a DC shift above. That is just a constant number--a constant of integration added to the data.

So please, do teach. Don't just laugh and insult. Any fool can do that. It takes an real scientist to explain.

As long as you will actually stop acting like everything I say on topics I use professionally must automatically be wrong and stupid just because I reject GW. I find this attitude quite common among GW advocates. People usually have to be burned to start being more careful. I know I have been burned in GW debates when I wasn't careful, so, I have been exactly where you are. A few months ago I got my head handed to me in one of the debates. I went back and started examining everything I believed about GW was wrong. Those months greatly improved my arguments. Maybe this will do that for you.

 

[thaumaturgy]

In response to Glenn's Laughing Friend I'd like to submit two last analyses of Time Series data. This is a dataset taken from the SAS Institute (JMP Statistics Software):

It shows data which has both a cyclical trend and a secular trend (steady increase). really, really hard to deny that.

The lower graph is the spectral density plot of same data. Note the little spike at or near the left-hand margin.

NOW, let's take the same data massaged to remove the secular trend:

NOTE: The secular trend is gone and now the detailed frequencies of the shorter wavelength data is much more apparent. AND there is no longer a "near-zero" spike in the data.

So, now, Glenn, since you've expended so much energy insulting me and you now have your friends "laughing" at me, do you care to comment?

(Or is it simply more fun to laugh at someone?)

 

[thaumaturgy]

THE STATE OF THE DEBATE

1. Thaumaturgy has made a large error in that he failed to note that the Michel Leroy "Siting Index" only accounts for positive bias. Hence Glenn's criticism that there is a possibility of a significant positive bias to the stations so far sampled in the U.S. by surfacestations.org is correct.

2. Thaumaturgy is still correct that surfacestation.org has only sampled 43% of the U.S. national sitings. The maps from surfacestation.org and the methodology of casting a net for "volunteers" indicates that their sampling protocol is not random, therefore the fact that they have found about 69% of the currently non-random sampled sites have said potential for positive bias means little in the way of statistical value.

The map itself on surfacestations.org is clearly biased for those areas with higher populations and hence higher probability of finding an installation at or near a "bad siting" criterion per the Leroy (1998) standard.

3. Glenn has also introduced a "red herring" of sorts in trumpetting the fact that surfacestations.org has sampled "100% of the California Stations". Again, that does not mean anything statistically as the temperature doesn't care one way or another whether there is a "state line" arbitrarily drawn by the U.S. government (which itself may introduce a bias) or not.

That is 54 stations, of which 38 have Leroy rating of 4 or 5.

So, are we really to abandon the entire U.S. grid because in a single arbitrary box in one of the most populace states in the U.S. 38 stations were found to have a potential for positive bias?

Again, I cannot stress enough that I was in grave error in not noting that the Leroy system only accounts for the potential of positive bias.

However, this is hardly a deathblow for the entirety of the data that support global warming.

Now, the problem for Glenn's side of the debate comes forth that:

Global warming evidence is not solely based on U.S. or U.K. or Chinese surface temperature measurements. It is from a number of lines of evidence which correlate among each other to verify the general trend.

NASA:

there are other potential sources of error, such as urban warming near meteorological stations, etc., many other methods have been used to verify the approximate magnitude of inferred global warming. These methods include inference of surface temperature change from vertical temperature profiles in the ground (bore holes) at many sites around the world, rate of glacier retreat at many locations, and studies by several groups of the effect of urban and other local human influences on the global temperature record. All of these yield consistent estimates of the approximate magnitude of global warming, which has now increased to about twice the magnitude that we reported in 1981. (SOURCE)

(emphasis added)

Now, it is highly unlikely that a stray air-conditioner unit would affect borehole temperature measurements and it would surely take a very large parking lot to significantly melt a glacier.

But further we don't even have to stay on the ground. Weather balloon data supports global warming trends as well:

Based on data from Angell's global network of 63 radiosonde stations, over the period from 1958 through 2005, the global mean, near-surface air temperature warmed by approximately 0.17°C/decade,(SOURCE)

Let's not limit ourselves to land or sky, let's look at ocean data:

A warming signal has penetrated into the world's oceans over the past 40 years. The signal is complex, with a vertical structure that varies widely by ocean; it cannot be explained by natural internal climate variability or solar and volcanic forcing, but is well simulated by two anthropogenically forced climate models. We conclude that it is of human origin, a conclusion robust to observational sampling and model differences. (SOURCE)

Now, I'll admit it's been about 17 years since I was on a research oceanographic cruise, but when I was out on the North Atlantic measuring gas exchange in sea water, I sure didn't see floating islands of air conditioner units or large parking structures. If I had you can be guaranteed I'd have been off that damn research ship in a second.

So, congrats, Glenn on pulling us off into anecdotal data by finding a likely non-random sampling of surface temperature sites with the potential for positive bias and dragging the conversation away from the mass of data that disconfirm your hypothesis.

It is still incumbent upon you to prove a statistically significant actual bias using proper statistics including true random (or near true random) sampling methodologies, and then explain how the various efforts by NASA, NOAA, and numerous international bodies to account for bias and error are somehow ineffective.

Then, please, feel free to get back to the little discussion of 95% Confidence Intervals vs standard deviation. (as you know sandard error of the mean is, by definition, smaller than the standard deviation unless you have only 1 data point. That's just the math:

SEM = s/sqrt(N)

Confidence Interval = t*s/sqrt(N)

(Now my math skills, having been maligned earlier may be in error on this but any time you divide a number by another number >1 means it gets smaller, so maybe you've found a way to get a standard error on the mean that is somehow larger than the standard deviation. I dunno.)

 

[thaumaturgy]

Glenn,
I recommend your friend who "laughed out loud" at the interpretation of the time-series analysis learn some statistics.

He might wish to speak with SAS. Perhaps in your dabbling in statistics you are familiar with SAS? (In case he's not they are one of the leading software companies for statistical measurement and analysis in the world. They are one of the most widely used stats software providers and have published numerous books on statistical analysis. You can read about them here)

Here's some information from SAS

The data are displayed as a thick black line in the top left plot. The periodogram of the data is shown as dots in the top right
panel. Note the exceptionally high periodogram values at low frequencies. This comes from the trend in the data. Because​

periodogram analysis explains everything in terms of waves, an upward trend shows up as a very long (low frequency) wave.(SOURCE)

(emphasis added)

I would dearly love you to addres this point because my pride is being pricked quite heavily on this matter. You have now found people to "laugh out loud" at my statements as if I were some common crackpot or fool.

I would hope that you will substantively respond because all information to the contrary, I don't think what I said is necessarily laughable, unless SAS and common statistical interpretation is "laughable".

 

[grmorton]

Glenn,
I recommend your friend who "laughed out loud" at the interpretation of the time-series analysis learn some statistics.

He might wish to speak with SAS. Perhaps in your dabbling in statistics you are familiar with SAS? (In case he's not they are one of the leading software companies for statistical measurement and analysis in the world. They are one of the most widely used stats software providers and have published numerous books on statistical analysis. You can read about them here)

Sigh, here we go again. My friend is the founder of about 6 different software companies in the oil industry. Each one he sold for millions. He is a very good mathematician. He started writing programs for Exxon when computers were just coming out. He decided he could make more money writing code for himself. He did. He was exceedingly successful.

Now, I would suggest that you understand what you really have. Today, in response to your question about phase spectra, I went searching for an image of phase spectra on the internet. They are exceedingly rare. because no one looks at them because no one understands intuitively what to do with them. Below are two examples.

The part marked phase, is the phase diagram that goes with the Fourier analysis. I made these myself because I couldn't find an example on the internet. Note that the phase data is just a wiggly line. I asked my friend if there was anyone who intuitively understood this part of the fourier. He said, he knew of no one.

Now, lets go back to your big peak in the low frequency. That big peak is the square of the amplitude of that frequency. The actual amplitude looks like the right side of each picture. It has both positives and negatives. Each white line marks 10 hz and the phase lines up with the amplitude to form a summation series which will give you the time series you started with. Note the negatives. I have colored each wiggle with a color fill for positive and negative values.

For your information, if I were to lay the original data down for these two transforms, the original data is very very very similar, yet one has a negative amplitude to start with and the other a positive in the amp spectra at low frequencies. And the phase spectra are different.



Now. How could one get a lot of amplitude in the low frequencies without having a secular trend? Well, in a long time series with sharp steps one will get low frequency energy--remember energy is amplitude squared, so that big red in the low frequency of one of the examples will cause a big amplitude in the power spectra, which you will erroneously interpret as a secular trend.

So, back to the question, how can one get lots of low frequency energy and still not have a secular trend? If the time series is long, then the low frequency component will go through its entire cycle and come back to the starting point without being part of a secular trend, but since it has lots of amplitude, it gives a peak in the power spectra.

This will be the last I speak of FFT. It is off topic, and frankly a miniscule issue here. You started off on this because you thought the Satellite data had a secular trend. But I haven't seen you answer the question of why, over the past 30 years, from start point to end your global temperature chart shows .6 - .7 deg C of change, but the satellite data only shows about a .2 deg C change. Why the discrepancy? Let's answer that first.


What you are calling the amplitude spectra, and what we in the oil industry call an amlitude spectra is really a power spectra--or amplitude squared, which physically is energy. Thus the charts are showing you energy at a particular frequency in the section. and the phase spectra tells you what phase the sine curve starts at. A big peak at the lower frequency ONLY means that those frequencies generaly start out close to being in phase, nothing more. ABSOLUTELY NOTHING MORE.

I find this utterly ridiculous for you to be claiming what you are claiming, yet you persist. You say you want me to teach and then you shut down and cease listening even on things you haven't understood.

Here's some information from SAS

(emphasis added)

I would dearly love you to addres this point because my pride is being pricked quite heavily on this matter. You have now found people to "laugh out loud" at my statements as if I were some common crackpot or fool.

Well, you are acting like one. It was like pulling teeth to get you to stop asserting that there was no bias when a thermometer was next to an air conditioner exhaust. Yes, you finally corrected your self, but you over and over kept ignoring my statements about how hard it is to have a heat source cool the thermometer. When you stop acting like a common crackpot and stop acting like you know things you don't know, then things can change. And you are the one who said that from -.2 to +.6 was about .5 or .6 deg C. That is not something that I would expect from a scientist. Science requires precision. So, if you want to stop asserting foolish claims, about the Fourier, then maybe we can move on to m ore productive discussions.

As I said, the only thing that makes for a peak ANYWHERE in a Fourier transform is when the sinusoids line up in phase. They can do that for a large number of reasons, none of which require a secular trend. I just remembered that a picture in my E&M textbook has an example which should convince you, if you are at all convinceable. But others will see the point even if you dont'. I will post it in my next note since I am editing this to adde these comments

I would hope that you will substantively respond because all information to the contrary, I don't think what I said is necessarily laughable, unless SAS and common statistical interpretation is "laughable".


[/quote]

 

[grmorton]

The following picture shows exactly why high amplitudes in the low frequency range don't mean a secular trend. This is the assertion of Thaumaturgy. As I mentioned, after I posted my last reply, I remembered that my old Electricity and Magnetism text had an example which used Fourier analysis to fit the summation of a series of sine waves to a box function. A box function is like this
._
|.|

The dots are only for spacing. I hope that picture comes out the same when I post it.

Now, a box function starts at zero, rises to some constant value, and then drops back to zero. There is NO secular trend. The starting and ending points are the same. But, Fourier analysis says that you can represent this box function as the infinite sum of a set of sine waves of different frequencies. The picture below shows how it is done. I have modified it to show the height of the amplitudes of the partial sum of terms.

The first term is the lowest frequency term. It ia a half wavelength with the period of the width of the box.

Lorraine and Corson say

"In order to have F=0 at y=0 we must have B=0; and in order to have V=0 at y=b we must have kb=n[pi] (n=1,2...)" p. 159."

Now, I have modified the original picture to show the red bar which is the amplitude of the LOWEST frequency. It is the biggest amplitude in the series and it is the lowest. Yet, a box function has no secular trend. If it starts at zero, then it ends at zero. I say this because frankly you seem to never understand what I write or are so determined to make me out to be wrong that you continually say silly things.

Now, the picture also shows the sum of the first 3 terms, 1, 2, and 3. Since we know what the first term looks like we can approximate the amplitude of terms 2 and 3 by finding the maximum difference between terms 1 and terms 1+2+3. That is the green bar. You will, I hope, agree that the green bar is shorter than the red bar. But the green bar is the sum of two terms and the red bar is the sum of one term. Thus, we should be able to logically conclude that terms 2 and three are smaller than term 1 in amplitude. If you don't agree with this then you are a common crackpot.

Now, the picture gives the sum of the first 10 terms, I have done the same thing, approximate the maximum sum of terms 4-10 with the brown bar. It is shorter than the green bar which is shorter than the red bar. That means that the terms 4-10 must all be smaller than term 1. This means that the first term has the largest amplitude, and thus the largest power, in a function which has no secular trend. I want you to be able to read that.

The same thing goes for the first 100 terms whose aggregate amplitudes are less than the amplitude of the first 10 terms. You can see the tiny wiggles in the final approximation to the box curve shape by the Fourier summation.

Now, one of the interesting things about this example is that all the frequencies start in phase at x=0, so you can't really say that the aggregate sums are messed up by being out of phase. They aren't.

Somehow, I think you will continue along this thread on FFT insisting that the high amplitude in the power spectra means a secular trend. YOu seem not to want to give up on this area. But you are quite wrong. While you may not respect my freind who laughed, he is one of the richest guys I know all based upon his ability to use mathematics.

with the posting of this picture, this strand of discussion on this thread must come to an end. I have clearly demonstrated that a high low frequency component doesn't have to mean a secular trend. A box function, even when it is not equal to zero, has no trend in the sense you are using the term.

Be mad at me if you will, but you have only yourself and your stubbornness on things about which you know little to blame.

Now, would you do me the favor of actually addressing my question about the Chinese data? Do you believe that that station in Sichuan China really had a yearly annual temperature below zero in 1971?

Would you please answer my point where I calculated the standard deviations of the temperatures in small areas and all the standard deviations (the error) was bigger than the proclaimed amount of global warming?

Would you answer anything other than staying on your quest to win a whizzing contest with me over Fourier?

 

[grmorton]

THE STATE OF THE DEBATE

1. Thaumaturgy has made a large error in that he failed to note that the Michel Leroy "Siting Index" only accounts for positive bias. Hence Glenn's criticism that there is a possibility of a significant positive bias to the stations so far sampled in the U.S. by surfacestations.org is correct.

2. Thaumaturgy is still correct that surfacestation.org has only sampled 43% of the U.S. national sitings. The maps from surfacestation.org and the methodology of casting a net for "volunteers" indicates that their sampling protocol is not random, therefore the fact that they have found about 69% of the currently non-random sampled sites have said potential for positive bias means little in the way of statistical value.

No it is not random in the statistical sense, but Thaumaturgy continues to avoid the question I asked. 100% of the California stations have been surveyed, and 35% have a 2 deg bias and 35% have a 5 deg plus bias. Can't we agree that the California contribution to global warming is crap? Shouldn't it be excluded?

The map itself on surfacestations.org is clearly biased for those areas with higher populations and hence higher probability of finding an installation at or near a "bad siting" criterion per the Leroy (1998) standard.

Well, in reality you haven't actually looked at the list of stations. Once again you are not doing your homework. LA and San Francisco and other large towns are NOT included in the global historical network, which I am using. All the towns are moderate in size. Colfax, California, one of the stations, has all of 1719 people according to a google search. What a hoot. you assume (of course without actually looking) that this is an urban area. Blythe California, another station had only 22,000 people. Once again, not a huge urban town. Cedarville is 220 people. Electra California has 774 souls. There are two figures on the internet for Eureka California, 26,000 or 42,000. That is a moderate town regardless. Brawley California 22,529.

Thaumaturgy, you need to do your homework before you make silly claims. I would have given you the benefit of the doubt, saying ok, you messed up with the Leroy scale, but you continue to do it over and over and over and over. And you don't give up. You make claims without actually checking to see if your claims are true. Do you think that Electra, California with a population of 774 is a big urban area? The tiny town my ranch is next to has about that many people. There are only 2 businesses in the town--a fertilizer plant and a restaraunt.

3. Glenn has also introduced a "red herring" of sorts in trumpetting the fact that surfacestations.org has sampled "100% of the California Stations". Again, that does not mean anything statistically as the temperature doesn't care one way or another whether there is a "state line" arbitrarily drawn by the U.S. government (which itself may introduce a bias) or not.

LOL. You can't even agree that the data from California should be excluded. How sad. You don't think that bad data should be excluded and you continually claim that I am the one who doesn't know statistics. What a hoot.

That is 54 stations, of which 38 have Leroy rating of 4 or 5.

So, are we really to abandon the entire U.S. grid because in a single arbitrary box in one of the most populace states in the U.S. 38 stations were found to have a potential for positive bias?

Once again, you are not very precise. I never said that we should abandon the US data based upon California. I would like you to document that bald faced assertion of yours or withdraw it. What I have asked is if we can agree that the Califronia data is crap. YOu are now side-stepping, (slip-slidin' away as Paul Simon sang) on this issue, by claiming something I never said. Put up the evidence where I said we should give up t he US data because California is bad or with draw that baseless bald-faced assertion.

Again, I cannot stress enough that I was in grave error in not noting that the Leroy system only accounts for the potential of positive bias.

And you are in grave error in claiming that the surveyed stations are in large population centers. You are in grave error in claiming that a large amplitude low frequency component of the Fourier transform means a secular trend. I just provided a counter example, which, no doubt, you will try to find some wiggle room on as you do on the question of the validity of California's data.

However, this is hardly a deathblow for the entirety of the data that support global warming.

Now, the problem for Glenn's side of the debate comes forth that:

Global warming evidence is not solely based on U.S. or U.K. or Chinese surface temperature measurements. It is from a number of lines of evidence which correlate among each other to verify the general trend.

Well, from start point to end point the Satellite data only rises by .18 deg C (and if one started one month later, and measured only to May 2008, one would see a decrease in tropospheric temperature over the 30 years. But the land data, subjected to parking lots, urban heat islands and other effects shows between .6 and .7 deg rise over the past 30 years. This discrepancy is something you need to explain. No doubt you will say it is my fault.

NASA:

(emphasis added)

It is nice to quote people but one should always check the data to see if the quote actually fits the data. Kind of like if I quoted you saying that there was a bias towards large population centers in the California data. I would hardly say a town of 774 or even 22,000 is a big city. I could quote you on that, but that wouldn't make it true.

Now, it is highly unlikely that a stray air-conditioner unit would affect borehole temperature measurements and it would surely take a very large parking lot to significantly melt a glacier.

I think I asked this before and I don't recall the answer. Maybe I forgot. but what are these borehole measurements you keep talking about. The thermal conductivity of either a borehole in the earth or a bore hole in a glacier totally excludes the measurement of global warming in a bore hole. In the earth, the yearly average temperature at a given site can be approximated by measuring the temperature about 3-6 feet down.

What bore holes are you talking about?

But further we don't even have to stay on the ground. Weather balloon data supports global warming trends as well:

I like this source. Take your pick of the numbers. The lottery is on:

"It is interesting to compare temperature trends from the relatively sparse 63-station radiosonde network to those from some of the other well-known global temperature records, e.g., the Microwave Sounding Unit (MSU) data obtained from NOAA satellites (Christy et al. 2000). While comparisons between the exact same atmospheric layers cannot be made, the computed linear trend for the MSU from its beginning year (1979) through 2005 for the lower tropospheric layer from the surface up to 8 km (about 350 mb) shows an increase of about 0.09°C/decade; whereas the Angell record for the 850-300 mb tropospheric layer over the same period shows no trend. The University of East Anglia data of Jones et al. (2001), derived from thousands of stations over the globe, indicate a global surface warming of 0.12°C/decade over Angell's full period of record (1958-2005) compared to 0.17°C/decade for the Angell data. For the years overlapping with the "satellite" period of record (1979-2005), the surface data of Jones et al. show an increasing trend of 0.17°C/decade; quite close to the 0.21°C/decade trend obtained from the 63-station network of Angell. " http://cdiac.esd.ornl.gov/trends/temp/angell/angell.html

Is it 0.09, 0.12, 0.17, or 0.21. Enquiring minds want to know.

Let's not limit ourselves to land or sky, let's look at ocean data:

Yes, let's look at the ocean data. Over the past 5 years the oceans have COOLED. You never responded to this picture either. YOu seem to have a selective response filter. Blue is cooling from 2003 to 2007. Yes, the oceans are still warmer than they were in 1990 but, we have more CO2 in the atmosphere in 2008 than we did in 2003 and the oceans are cooling. Why isn't the increase in CO2 warming the oceans? Remember, every blue dot is a grid element that cooled over the past 5 years.

edited to add. Notice that the land is in general heating up while the oceans are cooling. ONly on the land do we have hot parking lots, hot roof tops and hot cement beneath the thermometers, and only on land do we have air conditioners. I guess one alternative is that CO2 only sits over the land, and doesn't sit over the oceans, but I would find such an explanation ridiculous. Wouldn't you? (my confidence in your response has been shattered by some of the things I have seen you assert here).

Now, I'll admit it's been about 17 years since I was on a research oceanographic cruise, but when I was out on the North Atlantic measuring gas exchange in sea water, I sure didn't see floating islands of air conditioner units or large parking structures. If I had you can be guaranteed I'd have been off that damn research ship in a second.

So, congrats, Glenn on pulling us off into anecdotal data by finding a likely non-random sampling of surface temperature sites with the potential for positive bias and dragging the conversation away from the mass of data that disconfirm your hypothesis.

Well, Thaumaturgy, since I am the originator of this thread, and had certain purposes for it, it seems difficult to charge me with derailing my own thread. I would say that your stubborn refusal to see the data with the Fourier data is what has derailed things. But then, maybe you think you started this thread.

It is still incumbent upon you to prove a statistically significant actual bias using proper statistics including true random (or near true random) sampling methodologies, and then explain how the various efforts by NASA, NOAA, and numerous international bodies to account for bias and error are somehow ineffective.

And you think a heat source can cool a thermometer???? Even after your silliness with the Leroy scale? And you wonder why you might feel like you are being treated as a common crackpot. Do you never learn? Hot car engines parked next to thermometers will not result in the thermometer becoming cooler. If you can't understand that, nothing will help you.

Hot radiative cement beneath a thermometer will not cool that thermometer. If you can't understand that, nothing will help you.

Hot air conditioning exhaust blowing gently on a thermometer will not cool that thermometer. If you can't understand that, nothing will help you. I can do no better than this in proving that those stations will have a bias. If you think that they can be cooled by hot air blowing over them, I would just LOVE to hear the explanation. It is for this reason that I am beginning to think that you are what you worried about being treated as.

Then, please, feel free to get back to the little discussion of 95% Confidence Intervals vs standard deviation. (as you know sandard error of the mean is, by definition, smaller than the standard deviation unless you have only 1 data point. That's just the math:

I did post a standard devation post. I didn't see that you even responded. Maybe you can point me to it. I noted that the standard deviation of temperature measurments over very small areas was of the order of 2 deg F while the globe is supposed to have warmed only 1.1 deg. The signal of warming is smaller than the one standard deviation of error. That means, my friend, that you can't be sure that the globe has warmed. But getting you to actually respond to those issues rather than to respond with silliness on the Fourier transform seems to be rather difficult.

(Now my math skills, having been maligned earlier may be in error on this but any time you divide a number by another number >1 means it gets smaller, so maybe you've found a way to get a standard error on the mean that is somehow larger than the standard deviation. I dunno.)

I doubt your knowledge of FFT and I will now doubt your physics skills as well. How can you go about saying that I haven't proven a bias in the class 4 and 5 thermometers???? This is one of the most ridiculous things I have seen someone say. Hot air blowing across a thermometer doesn't bias it?????? How can that be? Are you really serious? Is your stubborness that great? Or are you the crank you don't want to be?

I would advise you to cut your losses. That is what I did when my head was handed to me in a previous GW debate on another list. The other guy was absolutely right and I was wrong on what I claimed at that time. I publically admitted that I was wrong, thus cutting my losses, saving my credibility for another day.

I seriously doubt that you will acknowledge what the Lorraine and Corson picture shows--that the a non-secular trending function can have a high amplitude in the low frequency range of the data. Somehow I doubt you will agree that California's data is crap. That is all I asked of you, and you can't even seem to doubt that much of your religion of GW. and I doubt that you will admit that the oceans are currently cooling down, as the data, derived from NOAA, clearly shows. See the picture below.

 

[Split Rock]

That is 54 stations, of which 38 have Leroy rating of 4 or 5.

No it is not random in the statistical sense, but Thaumaturgy continues to avoid the question I asked. 100% of the California stations have been surveyed, and 35% have a 2 deg bias and 35% have a 5 deg plus bias. Can't we agree that the California contribution to global warming is crap? Shouldn't it be excluded?

I have to say that, as a scientists, I am disgusted by this data. Especially since it is do to something as stupid as putting a thermometer in front of an air conditioning exhaust.

Glenn: I don't see the need to throw out all the CA data, however. What happens if the data with Leroy ratings of 4-5 are excluded?

 

[grmorton]

I have to say that, as a scientists, I am disgusted by this data. Especially since it is do to something as stupid as putting a thermometer in front of an air conditioning exhaust.

Glenn: I don't see the need to throw out all the CA data, however. What happens if the data with Leroy ratings of 4-5 are excluded?

Finally someone who actually cares about the data. Yippee! I was beginning to think that neither data, math, nor physics mattered to anyone on this list. Thanks a lot for this comment.

Your question is quite interesting. I wish I had thought of it. I should have but didn't. I will work on it and we will see. It might be Sunday. I gotta go to the ranch tomorrow but I will have a wee bit of time tonight and tomorrow morning to do it.

 

[grmorton]

I am downloading some of the stations I don't have and one of th e 'good' stations is Susanville CA. It is a class 1 station. Yep, a class 1 station. I don't know what the heck was going on in Susanville, but this is the temperature record that we are going to use to determine the global warming. I think it is utter incompetence. I know Thaumaturgy has doubted that, but if this is a good station..., lordy lordy!

After I posted the above, I downloaded Electra. It is even funnier, it is a class 2 station. I will post on it tonight. Split Rock, I think I can already answer your question, but I will go through the exercise to determine the proper answer to this question. I wouldn't want Thaumaturgy to accuse me of ignoring statistics.

 

[Tomk80]

I have to say that, as a scientists, I am disgusted by this data. Especially since it is do to something as stupid as putting a thermometer in front of an air conditioning exhaust.

Depends on what the original purpose of the thermometers was, does it not? "As a scientist" I often have to deal with data that is far from perfect, for example because it is collected by volunteers or because the data was never originally intended for the thing you want to use it for. This doesn't mean this data is useless, but it does mean you need to take into account it isn't perfect. From what I can gather, the data of the National Weather Services wasn't originally intended to specifically deal with long-term climate trends, nor do climate scientists have influence on how the data is gathered. If they don't own the stations, they can at best ask for suggestions to be taken into account. From what I can gather, the network is largely run by volunteers and the personnel involved is too low-staffed to check up on all sites. Next to that, a site may have been good to start with but deteriorated by local factors. Perhaps the thermometer was not placed next to the air vent but rather vice versa, for example?

The US Climate Reference Network has been specifically designed to measure long-term climate trends, but this network has only be operational since 2001, I don't know whether these have been used for comparison purposes already.

This does not make the data à priori useless, it just means that you need to take into account the error they can give. For example by giving less weight to data or by correcting the data of the worse stations with the help of the better stations before aggregating the data. As far as I understand, the latter is being done with the national weather stations. If you can estimate the error and direction of a measurement, you can use the measurement if you take this into account. So rather than rejecting the stations out of hand, you need to study the whole path from observation, via correction to eventual conclusions. Whatever has been discussed so far, the method of correction for differences between stations has been virtually ignored so far. Which to me, "as an epidmiologist" is "disgusting".

"As a scientist", I'm wondering what kind of scientist you are. Not meant denigrating, but I've often noticed differences between different "kinds" of scientists, often stemming from a lack of insight in the different problems different fields have. Watch a discussion between an epidemiologist and a toxicologist for hilarious effect. The difference between an epidemiologist who often has to use an estimate of exposure and effect over large groups to make statements for whole populations to a toxicologist who knows exactly how much of a certain substance and which effect he measured on his very low number of (human or animal) guinea pigs is extremely interesting.

 

[thaumaturgy]

So, maybe it is time for you to stop being so [bless and do not curse][bless and do not curse][bless and do not curse][bless and do not curse] sure of yourself on issues you are new to and be willing to actually listen and learn rather than constantly assuming that anyone who doubts GW must be a nutter. How about that. Then I will try to teach a bit.

Luke 6:31 (Before you preach at me, make sure you haven't done anything yourself that would have gotten me riled up a bit.

Because a peak at zero frequency is a constant shift. Don't confuse a ramp in frequency domain with a ramp in the time domain.

Well, pretty much everything I've shown from statistical literature indicates that that is not necessary the only explanation.

Perhaps you didn't read the stuff from SAS, so I'll repeat it:

The data are displayed as a thick black line in the top left plot. The periodogram of the data is shown as dots in the top right​

panel. Note the exceptionally high periodogram values at low frequencies. This comes from the trend in the data. Because

​

periodogram analysis explains everything in terms of waves, an upward trend shows up as a very long (low frequency) wave.(SOURCE)​

(emphasis added)​

Want some more proof? I took some data ginned up using a cyclic feature (seen in the top graph) then added a "zero-mean" linear trend (thanks to some advice from Chaim on this thread).​

Here's the form of the two trend equations:​

(for an arbitrary "f" and X=0-255, so you don't have to worry about 2[sup]n[/sup])​

(no trend)
Y = A*sin(X*f)​

(induced trend, no offset)
Y = A*sin(X*f)+(X-mean(X))​

The factor added in (X-mean(X)) is, by definition not a constant offset but a linear trend with increasing X,​

Then re-ran the time-series data and guess what? SAME THING.​

Top graphset data without trend, bottom graph set data with trend:

There's no "offset" in the data here. But there's still a massive peak at zero.​

In fact there's only one frequency of a cyclical pattern exactly where the equation was set to have one.​

 

[thaumaturgy]

I need to know:

Glenn, presented with the above data set, you can clearly see a linear trend increasing that is coupled with a cyclical trend.

I need to know how would you prove or disprove the obvious linear trend in the data that is unrelated to the cyclical data?

Go ahead, show me the "phase" diagram, just anything, I need to know.

Because I'm seeing a LOT of time series analyses and they run them and factor in or factor out the linear trends.

If you can show me you would prove or disprove a linear trend in data that has a cyclicity we can then revisit this issue in more detail as to whether the earlier "Global" temp data even has a linear trend in it.

Time Series analyses are extremely important to this conversation. Since Time Series are run for data all the time all across the globe, please tell me how you would describe the above data sets.

I'd be very interested to learn how you verify or falsify a linear trend in data that has a cyclic component in it. (Because such things do exist.)

Maybe that will help me prove or disprove my earlier contention on the Global Temperature data we were discussing many pages ago (that started all this).

(Please illustrate with a data set that shows both forms with and without trend and how you differentiate the data in a repeatable and robust manner).

 

[thaumaturgy]

EUREKA!!!

I have clearly demonstrated that a high low frequency component doesn't have to mean a secular trend.

THAT'S THE KEY! It doesn't have to mean a secular trend

BUT IT CAN

Finally I see where you are misreading my point! You have confused "necessary and sufficient" with "merely sufficient"!

Classic logic mistake.

I never said the spike at zero cannot mean a long wavelength function! I NEVER SAID THAT.

What I said was, the time series analysis treats secular trends as long wavelength functions.

Somehow, I think you will continue along this thread on FFT insisting that the high amplitude in the power spectra means a secular trend.

That is not the point of what I have been saying.

What I have been saying is perfectly reasonable in terms of fourier series.

The DETAILS OF MY STAND:

A "secular trend" in data which has, on top of it, cyclical trends will show up as a spike at (or very near) zero on a periodogram (a fourier transform of the data.

Precisely because a "secular trend" in data that is fourier transformed is treated exactly as if it were and extreeeeeeeemely long period wave.

That's all! Nothing more!

Now, imagine that the wavelength of this trend is something like 10 million years, but you only sample it for 30 years. The 10 million year long wavelength cycle will look like a linear trend within a 30 year window.

THAT is what I'm talking about, Glenn.

What you seem to be insisting on is that I am saying something more about the "secular trend".

I am not saying it is an ipso facto linear trend forever and ever and ever. For all I know (for all any of us knows) it might just be the start of an extremely long wavelength cycle.

In statistical analysis of TIME SERIES we are limited, as in all statistics, to forecasting within bounds that get wider the further we go out from the data.

I am not, nor have I ever stated in this debate that the trend will continue forever and ever upward. I don't know that.

What I can say is that within the data set so far analyzed there is a "secular trend" of increasing overall temperature that underlies other periodic functions on shorter time scales.

It could, indeed, be some giant geologic cycle. However here's why I think it is man-induced:

1. It corresponds to accelerating within humanity's industrial expansion

2. Humanity is pumping a known greenhouse gas into the atmosphere at a rate well in excess of a natural cycling (ie faster than it was sequestered)

3. We know, from isotopic analysis that the C in the CO2 recently ramped up at this alarming rate is from human combustion of fossil fuels.

Now, I put those 3 together with the clear evidence of an increase in overall temperature as evidenced by the "secular trend" the data clearly shows.

Take a look at my earlier "fake data" with and without a secular trend and tell me:

Why does the removal of a secular trend from the data which I put together by hand, result in the elimination of the spike at zero on the periodogram?

The ONLY CYCLIC TERMS IN MY DATA SET WAS A SIN WAVE WITH A PERIOD OF ABOUT 7 X-units.

When the secular trend was removed (x-mean(x)) it showed that clear 7 unit period in the frequency doman (~0.14)

While you may not respect my freind who laughed, he is one of the richest guys I know all based upon his ability to use mathematics.

I don't care how rich he is. Lol.

I suggest he take it up with SAS. I suspect they are a bit "richer" than your friend and based on their ability to use mathematics as well.


Oh and one last point in my defense:

I have never said that placing a thermometer next to a heat source will cause it to cool. You would be lying if you claimed that I have. You will note, as will everyone else, that what I actually said is that temperature bias can be induced both positively and negatively because of poor placement (supported by the citation of Peterson) and I merely confused that with the Leroy stance which only considered heat sources.
(Please re-read my post, apologize as you wish, or not, I don't expect you to be man enough to do that, but hope springs eternal.)

Be mad at me if you will, but you have only yourself and your stubbornness on things about which you know little to blame.

Maybe you need to take a logic class at some point. Learn what "necessary and sufficient" means.

All dogs are animals, not all animals are dogs.


NOW BACK TO THE STATISTICS TALK!

Would you please answer my point where I calculated the standard deviations of the temperatures in small areas and all the standard deviations (the error) was bigger than the proclaimed amount of global warming?

Gladly:

You took temperature data and averaged it for an entire year for sites in New York state? And you only found a standard deviation of +4 degrees on the worst year? That surprises me because I've lived in NY and I thought the temperature was more extreme than that.

Now, you focused on standard deviation. In the worst year, ~1966 you found a standard deviation of 4deg. I will assume this about 300 measurements (1 a day or so) which means that the MEAN TEMPERATURE in that year, is likely within 95% confidence equal to the mean +0.45

Do you see how I got that number?

I'll 'splain it to you:

95%CI = 1.96*s/sqrt(N)

s=standard deviation
1.96=z-score (95% of the population of a normal distribution is within 1.96 standard deviations of the mean, it would probably be slightly higher with t-score in place, so I rounded to 2 for this calculation)

N=number of observations.

The standard deviation you like to point out is actually about 68% of the population of the measurements collected. The CONFIDENCE INTERVAL is the likelihood that the "true" mean is somewhere in a much narrower band.

This is freshman statistics.

Please take your own advice and steer clear of topics you don't understand. Stop being so "[bless and do not curse][bless and do not curse][bless and do not curse][bless and do not curse] sure" of yourself. We are all prone to mistakes.

(Remember: those who avoid statistical analysis of data just feed Morton's Demon. Morton's Demon can live on a diet of anecdotal data)

 

[thaumaturgy]

I wanted a check on my thinking so I asked a PhD Statistician who is also a Six Sigma Master Blackbelt.

Here's what I asked him:

In the earlier e-mail you stated that you parsed out the "linear" trend as a zero peak in the periodogram of the RESIDUALS of the linear fit to the data.

I'm a bit confused. To help me understand this a bit more I ginned up a fake data set that has one cyclical component (a sin wave) with and without an overlayed linear trend (Y trend) which was generated by taking the sin function and then adding on an (X-mean(X)) factor to give it a nice linear trend.

I ran a time series on both and saw that nice big spike at zero for the time series data on Y-Trend.

Am I correct in the statement:

Linear trends in time-series data are often represented by a peak in the frequency periodogram at zero

Or am I missing something altogether here?

(Also, the residuals of the linear fit of the Y-Trend data shown here plot with the same sine wave frequency as the original data set, which is what I'd expect).

To which he responded thusly:

Yes, that is correct.

 

[Split Rock]

Depends on what the original purpose of the thermometers was, does it not? "As a scientist" I often have to deal with data that is far from perfect, for example because it is collected by volunteers or because the data was never originally intended for the thing you want to use it for. This doesn't mean this data is useless, but it does mean you need to take into account it isn't perfect. From what I can gather, the data of the National Weather Services wasn't originally intended to specifically deal with long-term climate trends, nor do climate scientists have influence on how the data is gathered. If they don't own the stations, they can at best ask for suggestions to be taken into account. From what I can gather, the network is largely run by volunteers and the personnel involved is too low-staffed to check up on all sites. Next to that, a site may have been good to start with but deteriorated by local factors. Perhaps the thermometer was not placed next to the air vent but rather vice versa, for example?

The US Climate Reference Network has been specifically designed to measure long-term climate trends, but this network has only be operational since 2001, I don't know whether these have been used for comparison purposes already.

This does not make the data à priori useless, it just means that you need to take into account the error they can give. For example by giving less weight to data or by correcting the data of the worse stations with the help of the better stations before aggregating the data. As far as I understand, the latter is being done with the national weather stations. If you can estimate the error and direction of a measurement, you can use the measurement if you take this into account. So rather than rejecting the stations out of hand, you need to study the whole path from observation, via correction to eventual conclusions. Whatever has been discussed so far, the method of correction for differences between stations has been virtually ignored so far. Which to me, "as an epidmiologist" is "disgusting".

"As a scientist", I'm wondering what kind of scientist you are. Not meant denigrating, but I've often noticed differences between different "kinds" of scientists, often stemming from a lack of insight in the different problems different fields have. Watch a discussion between an epidemiologist and a toxicologist for hilarious effect. The difference between an epidemiologist who often has to use an estimate of exposure and effect over large groups to make statements for whole populations to a toxicologist who knows exactly how much of a certain substance and which effect he measured on his very low number of (human or animal) guinea pigs is extremely interesting.

OK, Tom, if these stations were not setup for the studies they are being used for (or for accurate air temperature measurements in general), I can understand why the data is as error-prone as it seems to be. That makes it a statistical "dog's breakfast." Nevertheless, I would be more inclined to remove such data from the analysis, rather than try to correct it after the fact. Maybe there are issues with doing this?

I am a plant physiologist by training, and an agricultural physiologist by experience. And no, I don't normally deal with large-scale, global projects like this one. That is why I am letting others here lead the discussion.

 

[Split Rock]

I am downloading some of the stations I don't have and one of th e 'good' stations is Susanville CA. It is a class 1 station. Yep, a class 1 station. I don't know what the heck was going on in Susanville, but this is the temperature record that we are going to use to determine the global warming. I think it is utter incompetence. I know Thaumaturgy has doubted that, but if this is a good station..., lordy lordy!

What is wrong with this data? The linear regression shows a 3 degree increase in mean annual temperature over the time period measured. The cyclic nature of the data is apparent, but no one is arguing with that. We are talking about a Global Warming trend, not a warming for one station, or one state, or even one country. I applaud your efforts at routing out bad data, Glenn, but perhaps you are not seeing the forest because you are looking too hard at the individual trees.

After I posted the above, I downloaded Electra. It is even funnier, it is a class 2 station. I will post on it tonight. Split Rock, I think I can already answer your question, but I will go through the exercise to determine the proper answer to this question. I wouldn't want Thaumaturgy to accuse me of ignoring statistics.

Very good.

 

[Tomk80]

OK, Tom, if these stations were not setup for the studies they are being used for (or for accurate air temperature measurements in general), I can understand why the data is as error-prone as it seems to be.

IIRC, but this is from studying the temperature data long ago for a different project (correlating temperature changes with health outcomes) the data is normally used more for weather information. Specifically onset of seasons etc for farmers. Applications where a crude measure of temperature suffices.

That makes it a statistical "dog's breakfast."

Or statistician heaven My experience is that many statisticians love these kind of problems (or the statisticians that love these kind of problems go to this kind of work a lot of course, biased sampling through self-selection again).

Nevertheless, I would be more inclined to remove such data from the analysis, rather than try to correct it after the fact. Maybe there are issues with doing this?

It is an option. But with removing data you substitute one problem with another, because you now have missing data. And missing data, if not randomly distributed, can lead to biased results. So if you are removing data from your dataset, you still need to make an estimate of the effects this creates. My experience is that it is often better to have a number where you know the error, than a missing value. At least in health research it is becoming more and more common to put an estimate of a value in the place of a missing value instead of leaving the value open. This can reduce bias in your data set.

I am a plant physiologist by training, and an agricultural physiologist by experience. And no, I don't normally deal with large-scale, global projects like this one. That is why I am letting others here lead the discussion.

I see what you mean. I've worked with large datasets in the past, but never with trend analysis. And I really don't know enough about the data here to make accurate statements one way or the other, which is why I mostly read at this point.