February 2, 2012
Million Jews Found With Unknown Errors
The demographic duel continues. This is a reply to Bruce Phillips’ blog, but as my colleague Bruce has commented to me: It seem like too much “inside baseball” to be of general interest. So in the next blog I will try to put it into more popular terms.
Bruce, as social surveyors we are limited to the self-definition of respondents. Ultimately we have to classify survey respondents as the classify themselves.
I monitored an interview older gentleman answered no to whether he was a Holocaust survivor and all the other indicators pointed to yes. I called him to re-interview. His mother was Jewish and his father, a Wehrmacht officer and though he had suffered,his mother’s family perished, he still insisted and said he wasn’t a Holocaust survivor. I accepted that self-definition in the survey. As a social scientist it is not my role to “push further” only to convey information, to analyze and to comment.
I have an ethical responsibility to accurately collect and report the information conveyed to me by research subjects. If I routinely, and without informing the subject, ignore a normatively non-Jewish self-definition and self-description by shoehorning it into another category, such as Jewish, I am excluding subjects actual self-identities from consideration.
Research is inalienably and inevitably political (Letherby and Bywaters, 2006). DellaPergolas’s example that 12 million Americans of Jewish ancestry would qualify for Israeli citizenship under the Law of Return demonstrates it clearly. Ultimately, we as American Jews self-define who is an American Jew, not the Israeli government.
Is it a bit premature of Len Saxe to come out with a Jewish population figure that purports to be authoritative, almost census-like replacing a long delayed decennial National Jewish Population Study (NJPS)? He then seemingly disavows it, citing unknown error in the last paragraph of the study report:
I’m not quite sure that Saxe argued that Jews are more likely to refuse telephone surveys. They are not. Saxe’s reliance on a rather modest online Jewish sample, on which many of his estimation assumptions rest, is worrisome. Saxe’s use of a probability based online panel of 50,000 households yielding 1,300 Jewish households as the basis from which the sundry 150 surveys were demographically harmonized to yielded his magic 1.8 percent of the nation’s population being Jewish. This may be a case of error in the online Jewish online panel compounding error in the harmonized 150 surveys, of which the General Social Survey, I was waved off of using for Jewish population estimation by its principal investigator, Tom Smith.
The tiny sub-sample on which Saxe based his under age 18 population component would have yielded a Jewish population estimate of 7 million and so, inexplicably and modestly, he pulled back, almost 9 percent, to an estimate of 6.4 million Jews. Somewhere a population the size of Los Angeles’ Jewry was left arbitrarily on the cutting room floor as Saxe et. al. write:
I’m wondering what their present purpose, and even more worrisome, their future purpose is?
Telling was a table about families with children and their Jewish education sub-sample cells were so small that the category of 5 children is empty, but 4 and 6 were populated. Brandeis can’t improve their estimates without a more robust sample and its attendant higher cost. The sample of the Knowledge Networks online probabilistic panel yielded only 1,082 Jewish households, to attain something like the scale of a national or New York or LA study, 3 to 5 times the number of Jewish respondents would have to be obtained. This means GFK, the omnibus panel survey owner, would have to increase the size of their panel to 150 - 250 thousand, an unlikely investment of tens of millions of dollars.
I think Brandeis has hit a wall in terms their boast of finding a low-cost technique. They can only ride the low cost coattails of an omnibus survey, in this case Knowledge Networks, so far. So they jump off and hang on to the next trolley, the 150 cheaply available surveys that were done for someone else and and happen to ask “What is your religion?” This jump to the 150 surveys, harmonized to a tiny Jewish Knowledge Networks sample only compounds unknown error which both the original and harmonized surveys may share. The main source of error is that compared to non-Jews, Jews are enthusiastic survey takers. Jews will more likely vote and more likely to stay for the exit poll.
What make a more enthusiastic survey taker? Higher levels of education, income, health, leisure, news consumption, political involvement, occupational attainment. Sounds like Jews, doesn’t it? So what if Jews are enthusiastic survey takers? Well, that can bump up their share in the surveys Saxe used to estimate the Jewish population and may be enough to account for a million extra Jews. That would only be double the 600,000 Jewish kids he arbitrarily decided not to put in his final estimate.
Other sources of unknown error such as non-Jews calling themselves Jews may be greater than Saxe was able to ferret out from his online panel, from which he did take out two respondents (who said that they were Gentiles who believed in the religious principles of Judaism. Both were raised in religions other than Judaism and indicated that neither of their parents were Jewish). Another 79 respondents that Saxe was able to take out of his Knowledge Networks Jewish panel did not actually consider themselves to be Jewish by religion or other means. Such non-Jewish panel members were dropped by Saxe from analyses. Unfortunately, this controlling for non-Jewish false-positives was impossible to do from surveys long completed by others and on which Saxe heavily relies on for as a component of his Jewish population estimation methodology that yielded 6.4 million Jews. I believe that this created a non-random sample result from not taking into account the over-representation of actual Jews (and in the 150 secondary data surveys, both the over-representation of Jews
of non-Jews describing themselves as Jews) in the sampling procedures. Therefore, I question the accuracy of representation the entire Jewish population.
It’s almost been two weeks since I’ve put the Million Jews Mistake? blog online and I am hoping for a substantive reply from Len Saxe of Brandeis University.
Pini Herman, PhD. has served as Asst. Research Professor at the University of Southern California Dept. of Geography, Adjunct Lecturer at the USC School of Social Work, Research Director at the Jewish Federation of Greater Los Angeles following Bruce Phillips, PhD. in that position (I was recently notified that with 40,000 visitors this year the 15 year old study of the LA Jewish population was third most downloaded study from Berman Jewish Policy Archives in 2011) and is immediate past President of the Movable Minyan a lay-lead independent congregation in the 3rd Street area. Currently he is a principal of Phillips and Herman Demographic Research. To email Pini: firstname.lastname@example.org To follow Pini on Twitter: Follow @pinih