Starting today August 7th, 2024, in order to post in the Married Couples, Courting Couples, or Singles forums, you will not be allowed to post if you have your Marital status designated as private. Announcements will be made in the respective forums as well but please note that if yours is currently listed as Private, you will need to submit a ticket in the Support Area to have yours changed.
You keep saying this, but it's wrong. The value at zero means nothing at all. The zero frequency mode is the average value of the graph. It's not the value at zero that is any interest here, but instead the behavior near zero. You really should remove the zero mode from all such plots.There's no "offset" in the data here. But there's still a massive peak at zero.
Depends on what the original purpose of the thermometers was, does it not? "As a scientist" I often have to deal with data that is far from perfect, for example because it is collected by volunteers or because the data was never originally intended for the thing you want to use it for. This doesn't mean this data is useless, but it does mean you need to take into account it isn't perfect. From what I can gather, the data of the National Weather Services wasn't originally intended to specifically deal with long-term climate trends, nor do climate scientists have influence on how the data is gathered. If they don't own the stations, they can at best ask for suggestions to be taken into account. From what I can gather, the network is largely run by volunteers and the personnel involved is too low-staffed to check up on all sites. Next to that, a site may have been good to start with but deteriorated by local factors. Perhaps the thermometer was not placed next to the air vent but rather vice versa, for example?
The US Climate Reference Network has been specifically designed to measure long-term climate trends, but this network has only be operational since 2001, I don't know whether these have been used for comparison purposes already.
This does not make the data à priori useless, it just means that you need to take into account the error they can give. For example by giving less weight to data or by correcting the data of the worse stations with the help of the better stations before aggregating the data. As far as I understand, the latter is being done with the national weather stations. If you can estimate the error and direction of a measurement, you can use the measurement if you take this into account. So rather than rejecting the stations out of hand, you need to study the whole path from observation, via correction to eventual conclusions. Whatever has been discussed so far, the method of correction for differences between stations has been virtually ignored so far. Which to me, "as an epidmiologist" is "disgusting".
"As a scientist", I'm wondering what kind of scientist you are. Not meant denigrating, but I've often noticed differences between different "kinds" of scientists, often stemming from a lack of insight in the different problems different fields have. Watch a discussion between an epidemiologist and a toxicologist for hilarious effect. The difference between an epidemiologist who often has to use an estimate of exposure and effect over large groups to make statements for whole populations to a toxicologist who knows exactly how much of a certain substance and which effect he measured on his very low number of (human or animal) guinea pigs is extremely interesting.
What is wrong with this data? The linear regression shows a 3 degree increase in mean annual temperature over the time period measured. The cyclic nature of the data is apparent, but no one is arguing with that. We are talking about a Global Warming trend, not a warming for one station, or one state, or even one country. I applaud your efforts at routing out bad data, Glenn, but perhaps you are not seeing the forest because you are looking too hard at the individual trees.
I need to know:
Glenn, presented with the above data set, you can clearly see a linear trend increasing that is coupled with a cyclical trend.
I need to know how would you prove or disprove the obvious linear trend in the data that is unrelated to the cyclical data?
Go ahead, show me the "phase" diagram, just anything, I need to know.
Because I'm seeing a LOT of time series analyses and they run them and factor in or factor out the linear trends.
If you can show me you would prove or disprove a linear trend in data that has a cyclicity we can then revisit this issue in more detail as to whether the earlier "Global" temp data even has a linear trend in it.
Time Series analyses are extremely important to this conversation. Since Time Series are run for data all the time all across the globe, please tell me how you would describe the above data sets.
I'd be very interested to learn how you verify or falsify a linear trend in data that has a cyclic component in it. (Because such things do exist.)
Maybe that will help me prove or disprove my earlier contention on the Global Temperature data we were discussing many pages ago (that started all this).
(Please illustrate with a data set that shows both forms with and without trend and how you differentiate the data in a repeatable and robust manner).
grmorton said:The satellite data from Huntsville measures the temperature of the lower troposphere. As you can see the chart goes up and down but over 30 years, the tropospheric temperature is just about what it was 30 years ago, only a tiny tiny bit of net warming
my boldinggrmorton said:You didn't seem to notice that the ending temperature, in 2007 was about the same as the starting temperature in 1979. That is my point. I have no doubt that the temperature goes up and down, but a trend? not necessarily because today's low is not significantly higher than that of 30 years ago.The fact is that we have more CO2 in the atmosphere today than 30 years ago and we don't have a higher tropospheric temperature.
grmorton said:I will again upload the satellite data. From 1979 to 1997, there is no rise at all. Indeed, the zero line almost perfectly bifurcates the cyclical data. Then there is a bump with the very active solar cycles of the early 90s and early 2000s, then the temperature goes back to about where it was in 1979. I stand by this. This is NOT a linear phenomenon. The ups and down are NOT randomly distributed but cyclically distributed.
Please look at the data, not at your bias.
Thaumaturgy said:I will make the huge caveat that I have never done a "time series" analysis in the usual stats program I utilize.
Thaumaturgy said:Now, again, I am wholly new to the Fisher's kappa function but here's what JMP says about this function
{GRM--after quoting the handbook on the function Thaumaturgy then said}
This indicates to me that at 95% confidence (in fact at 98% confidence) I am reasonable in rejecting the hypothesis that this is more likely a "periodic" function within this time domain
And in that same post you wrotethaumaturgy said:I strenuously disagree. My linear regression has a p-value showing significance. YOU are under a burden to prove that the cyclicity is a better model.
Thaumaturgy said:Ask your stats friend to explain the Fishers' Kappa function. I would love to learn more about it. But from what I can tell, Fisher's Kappa indicates no such "cyclicity" among the noise to a 99% level of assurity.
grmorton said:First off, the satellite temperature data you have is a yearly time series, not a monthly. I used monthly. Thus, you miss out on the monthly periodicity. You still get a periodicity of 4 years rather than 64 months but I think that is because you are not using the monthly data but are clearly using annual data. That reduces the fidelity of the signal and impoverishes the frequency content. Beyond that, since I didn't do your analysis, I can't explain it.
Thaumaturgy said:I think I was mistaken about the Kappa function. It does show a statistical significance for cyclcity when it is low on the p-value.
No problem. We see from the graph that, as Glenn has pointed out, there is, indeed, cyclicity. AND it has a multi-year period. The residuals bear this out.
HOWEVER, from what I can tell the large peak at or near "zero" on the FREQUENCY graph, as well as the raw data graph itself, show a secular trend.
The way JMP models time-series is to assume the larger secular trend is actually just an extremely long-wavelength cyclicity I believe. Hence the "periodogram" in the lower left with an extremely long period peak of importance.
So we are back at square one. Indeed there is a cyclicity that is on a longer time scale than merely a seasonal as would be expected.
Thaumaturgy said:There's no "offset" in the data here. But there's still a massive peak at zero.
My bolding,Thaumaturgy said:I think I was mistaken about the Kappa function. It does show a statistical significance for cyclcity when it is low on the p-value.
No problem. We see from the graph that, as Glenn has pointed out, there is, indeed, cyclicity. AND it has a multi-year period. The residuals bear this out.
No, it doesn't mean it is useful. It doesn't mean it useless either, however. And I haven't seen you actually engage in that discussion yet. You reject the methods of making the data useful out of hand, without actually discussing it.Much of what you say is true. The data is collected by volunteers but that doesn't mean that the data is therefore useful for determining the climate of the past. It may be precisely because it was collected by volunteers that it is not useful.
You keep saying this, but it's wrong. The value at zero means nothing at all.
The zero frequency mode is the average value of the graph. It's not the value at zero that is any interest here, but instead the behavior near zero. You really should remove the zero mode from all such plots.
(emphasis added)The data are displayed as a thick black line in the top left plot. The periodogram of the data is shown as dots in the top rightperiodogram analysis explains everything in terms of waves, an upward trend shows up as a very long (low frequency) wave.(SOURCE)
panel. Note the exceptionally high periodogram values at low frequencies. This comes from the trend in the data. Because
Thaumaturgy said:In the earlier e-mail you stated that you parsed out the "linear" trend as a zero peak in the periodogram of the RESIDUALS of the linear fit to the data.
I'm a bit confused. To help me understand this a bit more I ginned up a fake data set that has one cyclical component (a sin wave) with and without an overlayed linear trend (Y trend) which was generated by taking the sin function and then adding on an (X-mean(X)) factor to give it a nice linear trend.
I ran a time series on both and saw that nice big spike at zero for the time series data on Y-Trend.
Am I correct in the statement:
Linear trends in time-series data are often represented by a peak in the frequency periodogram at zero
Or am I missing something altogether here?
(Also, the residuals of the linear fit of the Y-Trend data shown here plot with the same sine wave frequency as the original data set, which is what I'd expect).
PhD Statistician said:Yes, that is correct.
No, it doesn't mean it is useful. It doesn't mean it useless either, however. And I haven't seen you actually engage in that discussion yet. You reject the methods of making the data useful out of hand, without actually discussing it.
No, Thaumaturgy, I am becoming embarrassed by continuing to have to correct your amaturish mistakes. Let's review, and this will be my last reply to you on the Fourier issue. It is clear that you have a belief a priori to the data, so that you twist everything to fit your belief system. THERE MUST be a secular trend in the satellite data so you twist things to make it so. This FFT discussion started way back in post
In post 13 you binned the data into yearly bins and then said that the temperature rise seen in the satellite data was statistically significant.
In post 24 I said
(emphasis added).Statistician said:I think there is a reason to believe in a linear trend--it's called global warming. The opposite view is that the trend is just part of a longer cycle. The periodogram cannot distinguish between these two hypotheses. It can only distinguish cycles that repeat within the time span of data series.
Whatever the big picture really is, there is no doubt there is some kind of trend within the time span of your data.
In post 90 you rejected cyclicity in the satellite temperature data.
Finally in post 129, after standing on your kappa function, accusing me of not discussion statistics,
Then you changed the issue to try to say that the zero frequency was a secular trend. This is to try to maintain your belief that there is a strong secular trend in the satellite tropospheric temperature data in spite of it only going up at 1/3 the rate of your global temperature trend.
What rubbish that is. And you are still saying it.
Even the low frequency component doesn’t ensure a secular trend
, as I demonstrated with the box function, yet you continue to claim that Fourier analysis with high amp low frequnencies requires a secular trend.
It doesn’t. You clearly don’t know what the heck you are talking about
and you have had to back track numerous times, including once when you mis-read annual maximum temperature for being an annual temperature
You acknowledged that you hadn’t done time series analysis, but yet you act as if you know what you are doing. It is all bluster with you isn’t it?
You have made so many mistakes and errors, that I am beginning to get embarrassed to keep pounding you on them
The problem isn't near zero. You keep saying at zero. Yes, a trend results in strong low-frequency components. But the zero mode itself has nothing to do with the trend at all: the zero mode is just the average value of the data taken, since the zero-frequency mode is just a constant.Chalnoth, please re-read the various postings I've made from both the SAS Institute and a PhD statistician.
I will defer to experts on statistical time series analyses.
(YOu will note the same "peak at or near zero" shows up whether I have an offset or not in the data, as I showed in numerous previous postings).
so while a trend will always look like what you're seeing there, just seeing the strong spike in the spectrum near zero does not necessarily indicate an overall trend.
There is no question that there is a trend, but that comes primarily from looking at the time series data.
A really easy way to see this is to filter the data by removing all of the high-frequency information.
Well, no. You can take as an example a data series that is the sum of two linear trends, one increasing and one decreasing with equal slope. Overall there would be no trend, but the power spectrum would look nearly identical to the monotonically increasing trend.This is the "all dogs are animals, not all animals are dogs" part of the debate:
A spike at or near zero can indicate a secular trend (not a just according to me, but according to SAS and a PhD statistician). Indeed I have proven this in my data set by having and removing a secular trend in the data and showing the fourier transform spectral analyses of those.
One has the spike, one doesn't. The only thing that changes is the inclusion of a monotonic secular trend increasing.
This is where the filtering that I mentioned comes in. By filtering out the high-frequency noisy variation, the overall trend becomes more visible. It's essentially a somewhat more sophisticated way of binning the data. So, say, if you cut out the frequency scales with periods shorter than 3 years, the result is very similar to, at every point in the graph, taking the 3-year band surrounding that point, averaging it, and plotting the result.Agreed, but when looking at exceptionally noisy data in order to parse out the possibility of a secular trend in the data we have to rely on looking at the spectral analysis of the time-series data.
Well, more specifically, a trend will show up as a spike near zero (but not at zero: that's the average). The question is whether or not the spike indicates a trend or just long-wavelength variation.SAS and statisticians tell me, explicitly, that such a trend can show up as a spike at or near zero.
Well, not quite. There will be some ringing near the ends of the graph (a linear trend actually has significant components out all the way to the Nyquist frequency). But you're right, the result would be simple. It would be more interesting to perform on the real temperature data.That can be quite easily done without even running a filter on it. I can tell you the equation for that line is exactly:
Y=(X-MEAN(X))-MEAN{X-MEAN(X)}
That is a perfect line with mean Y value = 0. The spectral analysis of that plot comes out with only the spike at or near zero (as one would obviously expect).
Right, and what I'm saying is that if you're trying to demonstrate a secular trend, this is the sort of thing you do. The spectral analysis is interesting for picking out periodic variations, but not the trend. The trend shows up, but that feature that shows up can indicate things other than a trend as well.As you state, the analysis of the time-domain data shows that. A brute-force approach of a simple linear least-squares regression shows that the data has a statistically significant non-zero trend at 99.99% confidence.
Niggling question. What're the error bars on those little dots?
We use cookies and similar technologies for the following purposes:
Do you accept cookies and these technologies?
We use cookies and similar technologies for the following purposes:
Do you accept cookies and these technologies?