Discussion:
meaning of 'statistical variation'
Add Reply
Rich Ulrich
2024-11-02 17:32:01 UTC
Reply
Permalink
Lede -- Dead-heat poll results are astonishing – and improbable,
these experts say. Robert Tait

https://www.theguardian.com/us-news/2024/nov/02/what-polls-mean-so-far-trump-harris-election-voters


Of the last 321 polls in the battlegrounds, 124 - nearly 40% -
showed margins of a single point or less, the pair wrote.
Pennsylvania was the most “troubling” case, with 20 out of 59 polls
showing an exact tie, while another 26 showed margins of less than
1%.

This indicated “not just an astonishingly tight race, but also an
improbably tight race”, according to Clinton and Lapinski.

Large numbers of surveys would be expected to show a wider variety
of opinion, even in a close election, due to the randomness
inherent in polling. The absence of such variation suggests that
either pollsters are adjusting “weird” margins of 5% or more,
Clinton and Lapinski argued – or the following second possibility,
which they deemed more likely.

“Some of the tools pollsters are using in 2024 to address the
polling problems of 2020, such as weighting by partisanship, past
vote or other factors, may be flattening out the differences and
reducing the variation in reported poll results,” they write.

When the 'margin of error' is three points, then you expect the
observed results to VARY around the middle. Ignoring all
differences in method -- another survey, done one exactly the
same days by the same people is APT to differ by (say) three
points from the other. People with different methods and
sampling procedures would differ more.

If you flip a coin 1000 times, 500 heads is "most likely" but
if you get exactlly that number a bunch of times in a row,
someone is cheating. A range of results should center on
500, but 3% either way is EXPECTed to come up, fairly often.

The bell this rings for statisticians is the re-analyses done on
Gregor Mendel's data on Dominant/recessive genes. Mendel
wrote before modern statistics; he was a monk doing private
experiments; his work was buried in a minor publication, and
re-descovered years later. He published convincing data
showing ratios like 1-1, 3-1, 1-2-1 -- with TOO MUCH precision
for the number of plants he was growing.

Looking back, it seems that he very likely "fudged" his data in
one way or another. Reporting averages? Throwing out odd
results? His tables seem "unlikely" to have represented the
experiments as he described them.


I don't know how they selected their "321 polls" but I do know
that agencies with HIGH reputations are relatively few; and
the reputable ones do a pretty good job of "showing their
work", so I don't know what is going on here.

I can see how they may get more consistent results if they
do such things as "weighting by partisanship" -- this amounts
to a version of what is called "stratified sampling". Thus, they
are predicting changes in the overall outcome by looking at
changes in subgroups: Are suburban white college-educated
females changing their minds? - some of the surveys these
days are probably internet-based, and ask their volunteers
to give periodic responses, so there is a person-by-person
change measured, which could be precise.

What that LACKs is the randomness of selecting samples; and
where it potentially FAILs is that the generated outcome
makes iron-clad assumptions about turnout: This is our result
IF our percentages of assumed voters by category match
who votes. Unfortunately, WHO VOTEs seems to be the
big explanatory variable for WHO WINs in recent elections.

Nate Silver, of FiveThrtyEight, is the most prestigious of US
poll interpreters. Last week, he gave his statistical opinion,
that, despite all the close calls in polls, there is about a 60%
chance (IIRC) that one candidate or the other will win at
least 6 of the 7 swing states. Sounds about right to me.
Oh, he said that the data (last week) gave Trump about
a 53% chance of winning.
--
Rich Ulrich
David Jones
2024-11-02 23:36:09 UTC
Reply
Permalink
Post by Rich Ulrich
Lede -- Dead-heat poll results are astonishing – and improbable,
these experts say. Robert Tait
https://www.theguardian.com/us-news/2024/nov/02/what-polls-mean-so-far-trump-harris-election-voters
Post by Rich Ulrich
Of the last 321 polls in the battlegrounds, 124 - nearly 40% -
showed margins of a single point or less, the pair wrote.
Pennsylvania was the most “troubling” case, with 20 out of 59 polls
showing an exact tie, while another 26 showed margins of less than
1%.
This indicated “not just an astonishingly tight race, but also an
improbably tight race”, according to Clinton and Lapinski.
Large numbers of surveys would be expected to show a wider variety
of opinion, even in a close election, due to the randomness
inherent in polling. The absence of such variation suggests that
either pollsters are adjusting “weird” margins of 5% or more,
Clinton and Lapinski argued – or the following second possibility,
which they deemed more likely.
“Some of the tools pollsters are using in 2024 to address the
polling problems of 2020, such as weighting by partisanship, past
vote or other factors, may be flattening out the differences and
reducing the variation in reported poll results,” they write.
When the 'margin of error' is three points, then you expect the
observed results to VARY around the middle. Ignoring all
differences in method -- another survey, done one exactly the
same days by the same people is APT to differ by (say) three
points from the other. People with different methods and
sampling procedures would differ more.
If you flip a coin 1000 times, 500 heads is "most likely" but
if you get exactlly that number a bunch of times in a row,
someone is cheating. A range of results should center on
500, but 3% either way is EXPECTed to come up, fairly often.
The bell this rings for statisticians is the re-analyses done on
Gregor Mendel's data on Dominant/recessive genes. Mendel
wrote before modern statistics; he was a monk doing private
experiments; his work was buried in a minor publication, and
re-descovered years later. He published convincing data
showing ratios like 1-1, 3-1, 1-2-1 -- with TOO MUCH precision
for the number of plants he was growing.
Looking back, it seems that he very likely "fudged" his data in
one way or another. Reporting averages? Throwing out odd
results? His tables seem "unlikely" to have represented the
experiments as he described them.
I don't know how they selected their "321 polls" but I do know
that agencies with HIGH reputations are relatively few; and
the reputable ones do a pretty good job of "showing their
work", so I don't know what is going on here.
I can see how they may get more consistent results if they
do such things as "weighting by partisanship" -- this amounts
to a version of what is called "stratified sampling". Thus, they
are predicting changes in the overall outcome by looking at
changes in subgroups: Are suburban white college-educated
females changing their minds? - some of the surveys these
days are probably internet-based, and ask their volunteers
to give periodic responses, so there is a person-by-person
change measured, which could be precise.
What that LACKs is the randomness of selecting samples; and
where it potentially FAILs is that the generated outcome
makes iron-clad assumptions about turnout: This is our result
IF our percentages of assumed voters by category match
who votes. Unfortunately, WHO VOTEs seems to be the
big explanatory variable for WHO WINs in recent elections.
Nate Silver, of FiveThrtyEight, is the most prestigious of US
poll interpreters. Last week, he gave his statistical opinion,
that, despite all the close calls in polls, there is about a 60%
chance (IIRC) that one candidate or the other will win at
least 6 of the 7 swing states. Sounds about right to me.
Oh, he said that the data (last week) gave Trump about
a 53% chance of winning.
Of course I know nothing about this, but one possible explanation is
that the results are all obtained by some version of "shrinkage", as in
https://en.wikipedia.org/wiki/Shrinkage_(statistics)
... possibly they are all shrinking estimates towards a 50-50 result.
But if they did do this, presumably they would have to adjust the usual
"margin of error" in some way.
Rich Ulrich
2024-11-03 22:50:02 UTC
Reply
Permalink
On Sat, 2 Nov 2024 23:36:09 -0000 (UTC), "David Jones"
Post by Rich Ulrich
Post by Rich Ulrich
Lede -- Dead-heat poll results are astonishing – and improbable,
these experts say. Robert Tait
https://www.theguardian.com/us-news/2024/nov/02/what-polls-mean-so-far-trump-harris-election-voters
Post by Rich Ulrich
Of the last 321 polls in the battlegrounds, 124 - nearly 40% -
showed margins of a single point or less, the pair wrote.
Pennsylvania was the most “troubling” case, with 20 out of 59 polls
showing an exact tie, while another 26 showed margins of less than
1%.
This indicated “not just an astonishingly tight race, but also an
improbably tight race”, according to Clinton and Lapinski.
Large numbers of surveys would be expected to show a wider variety
of opinion, even in a close election, due to the randomness
inherent in polling. The absence of such variation suggests that
either pollsters are adjusting “weird” margins of 5% or more,
Clinton and Lapinski argued – or the following second possibility,
which they deemed more likely.
“Some of the tools pollsters are using in 2024 to address the
polling problems of 2020, such as weighting by partisanship, past
vote or other factors, may be flattening out the differences and
reducing the variation in reported poll results,” they write.
When the 'margin of error' is three points, then you expect the
observed results to VARY around the middle. Ignoring all
differences in method -- another survey, done one exactly the
same days by the same people is APT to differ by (say) three
points from the other. People with different methods and
sampling procedures would differ more.
If you flip a coin 1000 times, 500 heads is "most likely" but
if you get exactlly that number a bunch of times in a row,
someone is cheating. A range of results should center on
500, but 3% either way is EXPECTed to come up, fairly often.
The bell this rings for statisticians is the re-analyses done on
Gregor Mendel's data on Dominant/recessive genes. Mendel
wrote before modern statistics; he was a monk doing private
experiments; his work was buried in a minor publication, and
re-descovered years later. He published convincing data
showing ratios like 1-1, 3-1, 1-2-1 -- with TOO MUCH precision
for the number of plants he was growing.
Looking back, it seems that he very likely "fudged" his data in
one way or another. Reporting averages? Throwing out odd
results? His tables seem "unlikely" to have represented the
experiments as he described them.
I don't know how they selected their "321 polls" but I do know
that agencies with HIGH reputations are relatively few; and
the reputable ones do a pretty good job of "showing their
work", so I don't know what is going on here.
I can see how they may get more consistent results if they
do such things as "weighting by partisanship" -- this amounts
to a version of what is called "stratified sampling". Thus, they
are predicting changes in the overall outcome by looking at
changes in subgroups: Are suburban white college-educated
females changing their minds? - some of the surveys these
days are probably internet-based, and ask their volunteers
to give periodic responses, so there is a person-by-person
change measured, which could be precise.
What that LACKs is the randomness of selecting samples; and
where it potentially FAILs is that the generated outcome
makes iron-clad assumptions about turnout: This is our result
IF our percentages of assumed voters by category match
who votes. Unfortunately, WHO VOTEs seems to be the
big explanatory variable for WHO WINs in recent elections.
Nate Silver, of FiveThrtyEight, is the most prestigious of US
poll interpreters. Last week, he gave his statistical opinion,
that, despite all the close calls in polls, there is about a 60%
chance (IIRC) that one candidate or the other will win at
least 6 of the 7 swing states. Sounds about right to me.
Oh, he said that the data (last week) gave Trump about
a 53% chance of winning.
Of course I know nothing about this, but one possible explanation is
that the results are all obtained by some version of "shrinkage", as in
https://en.wikipedia.org/wiki/Shrinkage_(statistics)
... possibly they are all shrinking estimates towards a 50-50 result.
But if they did do this, presumably they would have to adjust the usual
"margin of error" in some way.
Here is another commentary on the same thing.
https://www.emptywheel.net/2024/11/03/male-pollsters-shocked-shocked-when-a-woman-pollster-discovers-women-voters/?fbclid=IwY2xjawGU1ChleHRuA2FlbQIxMQABHXSltRouQ_w_bulSgPL75SATJWOXBb2NZFhDmk3_5vXMG_wWdKCVNw5UhQ_aem_fJ-vwR17bW3o71gQRR9g6Q

What this article calls 'herding' does sound like it might be
'shrinkage' -- though, shrinkage is a particular procedure.
The author here partly seems to blame Nate Silver for making
everyone so cautious, though I don't know where his hand
comes into it. The article suggests, over-correcting for previous
errors in missing the pro-Trump vote.

New news: a well-respected pollster has newly measured a
bigger lead for Harris in Iowa, and that has stimulated more
reaction.

The people pushing for turnout have been a bigger presence
than I've seen before -- three phone calls, a door-knocking,
and a thingie hung on my doorknob. I'm in Pennsylvania, which
is targeted as a swing state with 19 electors.
--
Rich Ulrich
Rich Ulrich
2024-11-11 01:08:52 UTC
Reply
Permalink
On Sun, 03 Nov 2024 17:50:02 -0500, Rich Ulrich
Post by Rich Ulrich
Nate Silver, of FiveThrtyEight, is the most prestigious of US
poll interpreters. Last week, he gave his statistical opinion,
that, despite all the close calls in polls, there is about a 60%
chance (IIRC) that one candidate or the other will win at
least 6 of the 7 swing states. Sounds about right to me.
Trump wins 7 of 7 swing states, now that Arizona has been 'called'.

Turnout: Trump received ABOUT as many votes as he did in 2020,
probably more than 50% of the total votes -- a bit of a surprise
since "53%" of prospective voters say they dislike him. (Lesser
of two evils, I guess.)

Since Biden won by millions, that means that Harris received
notably fewer than Biden had -- turnout mattered. Trump's
edge was up (they say) even more in the states that were
not the swing states (and did not receive saturation-levels of
TV advertising and local visits).
--
Rich Ulrich
Peter Moylan
2024-11-11 02:12:01 UTC
Reply
Permalink
Post by Rich Ulrich
On Sun, 03 Nov 2024 17:50:02 -0500, Rich Ulrich
Post by Rich Ulrich
Nate Silver, of FiveThrtyEight, is the most prestigious of US
poll interpreters. Last week, he gave his statistical
opinion, that, despite all the close calls in polls, there is
about a 60% chance (IIRC) that one candidate or the other will
win at least 6 of the 7 swing states. Sounds about right to
me.
Trump wins 7 of 7 swing states, now that Arizona has been 'called'.
Turnout: Trump received ABOUT as many votes as he did in 2020,
probably more than 50% of the total votes -- a bit of a surprise
since "53%" of prospective voters say they dislike him. (Lesser of
two evils, I guess.)
Since Biden won by millions, that means that Harris received notably
fewer than Biden had -- turnout mattered. Trump's edge was up (they
say) even more in the states that were not the swing states (and did
not receive saturation-levels of TV advertising and local visits).
Opponents of compulsory voting have often claimed that, if everyone
votes, the results will be distorted by the votes of the unintelligent
and of the "don't care" voters.

Is the latest US result an illustration of what happens when only the
intelligent voters vote?
--
Peter Moylan ***@pmoylan.org http://www.pmoylan.org
Newcastle, NSW
Silvano
2024-11-11 09:19:50 UTC
Reply
Permalink
Post by Peter Moylan
Is the latest US result an illustration of what happens when only the
intelligent voters vote?
Thanks for the laugh. OTOH, when I look at the latest regional elections
here in Germany and see that more people voted than in the previous
regional elections and at the same time many more people voted a party
which has become fully neofascist (AfD), I'm not so sure that compulsory
voting is the solution.

Democracy has an intrinsic problem and I have no idea how to solve it:
it's almost impossible to keep it when the absolute majority votes
against democracy.
J. J. Lodder
2024-11-11 19:13:34 UTC
Reply
Permalink
Post by Silvano
Post by Peter Moylan
Is the latest US result an illustration of what happens when only the
intelligent voters vote?
Thanks for the laugh. OTOH, when I look at the latest regional elections
here in Germany and see that more people voted than in the previous
regional elections and at the same time many more people voted a party
which has become fully neofascist (AfD), I'm not so sure that compulsory
voting is the solution.
it's almost impossible to keep it when the absolute majority votes
against democracy.
Know already to the ancient Greeks of course.
They noted that democracy, oligarchy, and tyranny
tended to follow each other.

I'm afraid this no longer works.
Todays tyrannies tend to be long-lived,
because they control the means of communication,

Jan
Rich Ulrich
2024-11-13 07:09:18 UTC
Reply
Permalink
Post by J. J. Lodder
Post by Silvano
Post by Peter Moylan
Is the latest US result an illustration of what happens when only the
intelligent voters vote?
Thanks for the laugh. OTOH, when I look at the latest regional elections
here in Germany and see that more people voted than in the previous
regional elections and at the same time many more people voted a party
which has become fully neofascist (AfD), I'm not so sure that compulsory
voting is the solution.
it's almost impossible to keep it when the absolute majority votes
against democracy.
Know already to the ancient Greeks of course.
They noted that democracy, oligarchy, and tyranny
tended to follow each other.
I'm afraid this no longer works.
Todays tyrannies tend to be long-lived,
because they control the means of communication,
Are you thinking of the god-emperor of Thailand? You can go
to jail if you say bad things about him or his government.

The Soviet Union finally lost control of the "means of communication"
when floppy disks and PCs became widespread, but copying
machines had undermined them before that. Samisdat?

When the original tyrant is still in power, I don't feel it fits any
historical perspective, to say the tyranny is "long-lived."
Are there any information-controlling tyrannies that pre-date
WW II?
--
Rich Ulrich
Athel Cornish-Bowden
2024-11-13 08:47:52 UTC
Reply
Permalink
Post by Rich Ulrich
Post by J. J. Lodder
Post by Silvano
Post by Peter Moylan
Is the latest US result an illustration of what happens when only the
intelligent voters vote?
Thanks for the laugh. OTOH, when I look at the latest regional elections
here in Germany and see that more people voted than in the previous
regional elections and at the same time many more people voted a party
which has become fully neofascist (AfD), I'm not so sure that compulsory
voting is the solution.
it's almost impossible to keep it when the absolute majority votes
against democracy.
Know already to the ancient Greeks of course.
They noted that democracy, oligarchy, and tyranny
tended to follow each other.
I'm afraid this no longer works.
Todays tyrannies tend to be long-lived,
because they control the means of communication,
Are you thinking of the god-emperor of Thailand? You can go
to jail if you say bad things about him or his government.
The Soviet Union finally lost control of the "means of communication"
when floppy disks and PCs became widespread, but copying
machines had undermined them before that. Samisdat?
When the original tyrant is still in power, I don't feel it fits any
historical perspective, to say the tyranny is "long-lived."
Are there any information-controlling tyrannies that pre-date
WW II?
Yes. Stalin came to power by exactly that method. He realized while
Lenin was still alive that the Secretary of the Communist Party was
more powerful than the President or anyone else because he could
control the flow of information, so all the minor officials only knew
what he wanted them to know.
--
Athel -- French and British, living in Marseilles for 37 years; mainly
in England until 1987.
Loading...