Processing research that's based on shaky assumptions?
Hi Zettelkasten friends,
How do you process research that's based on shaky assumptions? By "shaky" I mean that in the middle of reading, you figured that the authors were either assuming that a specific concept is true, or are implying that the specific concept applies to their study's context.
So for example, a lot of other studies are based on Zeigarnik Effect, but other studies on the same concept have failed to replicate the phenomenon. Or implying that the "need for cognitive closure" applies to the "task" context, even when the cited paper refers to closure in the context of beliefs.
Their results would fit nicely into their assumptions and hypotheses, but the assumptions themselves don't hold up based on the other papers.
So if you were in my place, do you think the experiments would still be valuable when you interpret the results yourself? How would you process them?
I'm actually frustrated right now because after processing Sophie Leroy's famously cited paper on attention residue, I discovered that most of her citations in "need for cognitive closure" were not in the context of attention, but rather on beliefs and the act of finding more information to attain certainty. Specifically, she asserts that:
As a result, compared to people who are motivated to reach cognitive closure, people who are not motivated to reach cognitive closure are more likely to keep thinking about a task even after it has been finished.
...even though the papers she cited before that statement did NOT refer to any context of a "task" at all. I'd feel like I've just wasted hours reading it, if it weren't for the tangentially made notes.
Anyways, here's the TL;DR version
- How do you process research that's based on shaky assumptions?
- If the study was experimental/observational, do you think the experiments would still be valuable when you interpret the results yourself — despite knowing that the assumptions were shaky?
Side note: If you have any resource recommendations on guides re: how to process experimental studies better, that'd be most helpful!
Howdy, Stranger!
Comments
My personal view: ALL information is "valuable" -- or nearly all of it. In some cases it is worth recording what you (or others) think of it as part of your notes. Bearing in mind that your opinion (and that of others) may change. The only place where you can find absolute unchanging "truths" is in religion. For the rest, the journey is perhaps more important than the destination. I find myself thinking of these lines from Eliot's Four Quartets:
We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.
I divide claim and evidence for it. A claim is more akin to an abstract idea (like the Zeigarnik-Effect) and the evidence is attached to it.
If I use the Zeigarnik-Effect (I used it before I encountered its poor replicability) I'd be transparent on the available evidence for it.
I am a Zettler
Oh, that was simpler than I thought. Can you give a concrete example of this? In my session earlier, I did this:
I don't know if it's right, but since Leroy claims that "attention residue" is a thing, then probably:
Experience so far using "layers of evidence": I found that I was able to avoid a bit of confirmation bias from the author, and foggy causal claims, i.e. author claiming "time pressure causes more confidence of completion" instead of "time pressure caused people to work faster, which led to more actual completed work — as a result, they felt more confident."
Unrelated, but what you just said reminds me of the phrase "the only thing that's certain in life is death"... Anyway, maybe it is relevant — in the angle that scientific claims have the possibility to be proven wrong since there's no real absolute, unchanging "truths."
1+1 will always equal 2. Seems absolute and unchanging to me.
I was overloading your operator? There is more to mathematics then algebraic logic.
my first Zettel uid: 202008120915
https://en.wikipedia.org/wiki/2_+_2_=_5
I am a Zettler
A litany of non-mathematical, aka art and literature examples of the symbolic meaning of addition and it's various interpretations isn't proof of the mathematical corollary. Nor is relying on the old "but in binary 1 plus 1 equals 10."
the core question is whether absolute truth can exist or not.
This reminds me of a popular figure of speech: "counting one and one together". It is used to empathize that something is based on common sense and does not need explanation.
Since common self does not question itself, absolute truth can be achieved.
In contrast, a logic system is never flawless:
formal language uses an alphabet (for example, natural integer) to represent knowledge and a grammar (for example, addition) to act on that knowledge. By that we can express an argument with which we achieve unambiguous truth, but not absolute truth.
How would one deal with ambiguous conclusions in a Zettelkasten? Should we ignore them altogether? Should we create multiple, unambiguous systems around them? These are challenges we see ourself confronted with once we don't treat every information as given.
my first Zettel uid: 202008120915
Although I agree with much of the spirit of what @MartinBB said above, I am going to disagree with the part that says "all information is valuable". I am eager to ruthlessly exclude information from my note system, for various reasons, although I understand that others would be more inclusive for other reasons.
@alcantal said:
I see two criteria here: (1) truth and (2) applicability or relevance.
(1) Obvious falsity can be good reason for rejecting information. But as I think @zk_1000 was saying, some degree of falsity or error is inevitable—or as Martin Mahner and Mario Bunge said, "science provides the best possible factual knowledge, even though it may, and does, in fact, contain errors". There are ways to mark partial truth or error. For example, if one's notes are as atomic as they should be and if, as we were recently discussing elsewhere, one thinks in terms of some taxonomy or ontology like IBIS (issues/questions, positions/answers, and arguments) or The Craft of Research (topics, questions, problems, claims, reasons and their warrants, evidence, objections, and responses), then one can connect any position/claim or evidence to a web of counterarguments and critical questions. So one might choose to include information that is obviously false or erroneous so as to argue against it and try to fix it, or one can argue against apparently true information (conceptually or empirically testing it) to try to discover falsity or error. As I suggested elsewhere, one can think of this in terms of defeasible reasoning: the note system is part of an informal defeasible reasoning system.
(2) Irrelevance is, for me, a very important reason to reject information. If it is irrelevant, it doesn't go into my note system. Relevance is an intuitive judgment that I make according to my purposes, which are represented in my note system as topics and issues/questions and sub-issues/questions. (Make sure the questions in your note system are detailed enough to meaningfully evaluate the relevance of information!) I could change my mind later about the relevance of some information if the balance of considerations changes as my purposes change. For my note system, relevance is a more important filter than truth, because partially true or false information can be argued against in the note system and still be valuable, but irrelevant information is just worthless noise. I can't think of a situation when I realized that information was irrelevant after it entered my note system (so my relevance filter must be working), but I would certainly remove information if I reached the conclusion that it is irrelevant.
@alcantal said:
I would suggest reviewing a good research methods textbook in any of the fields that you are working in; extract from it more specific criteria that you can use to criticize questions, positions/claims, models (which I think of as a complex kind of position/claim), reasons, and evidence, or else to reject them as irrelevant; and use those criteria when you first evaluate sources, before you spend much time processing them. Also, constantly keep seeking to deepen your personal theory of knowledge, and use improvements in your personal theory of knowledge to improve your note system.
@Andy I think we all inevitably (and often unconsciously) see these things through the lens of our field of interest. I can certainly see why someone would use relevance as a criterion if their notes archive was restricted to a particular field or subject area. My main areas of interest are psychology/psychotherapy and history, with additional interests in literature, the visual arts, software that I use, and so forth. For me, it is perfectly possible that some observation about computer software will also be illuminating about psychology, or that psychology will illuminate something in history. I see connections all the time, and often the more interesting ones are those that cross boundaries in some way or another. I hope this explains why I see almost all information as being valuable. I am an inveterate boundary-crosser. Hell, I even lived in another country for ten years!
Just as an example of boundary-crossing, and how illuminating it can be, I offer this article: clausewitz.com/item/Beyerchen-ClausewitzNonlinearityAndTheUnpredictabilityOfWar.htm. I am not a scientist and knew nothing about chaos theory, but as soon as I read this article about how it might apply to history and warfare I was off looking for videos of jointed pendulums and the like. It opened up a whole new way of thinking about certain phenomena, including aspects of human behaviour.
In a sense I have a huge problem because not much is irrelevant to what I am interested in -- or I could say that I often see how something might potentially be relevant to my interests. It has led to a big pile of material collected over the decades, but the Zettelkasten method is helping me to connect things that should be connected.
@MartinBB said:
What I mean by relevance is not exactly interest. I am interested in all kinds of information (I am an "inveterate boundary-crosser" too in that way) but not all of it goes into my note system. Nor does relevance mean that some information is vaguely associated with a topic. Information is relevant if it will help answer a specified question or solve a specified problem, or will help question an existing answer or solution to a specified question or problem.
This definition of relevance reminds me of what the mathematician Richard Hamming said in "You and Your Research":
On my computer, I have lots of information stored outside of my note system. In Hamming's terms, that information is not in my note system because I have not seen how it can be used to "attack my problems".
The artificial intelligence in one of the software applications that I use (DEVONthink) can suggest, for any note, other information, including information outside of my note system, that is topically similar to the note based on language. Of course, my own brain can make such suggestions too, but sometimes it is interesting to check and see if the computer can help. Similarity, however, is no guarantee of relevance.
How does one know something is relevant to solving a problem until they KNOW it's actually relevant? Isn't that one of the beauties of the ZK wherein one stumbles upon new ideas (debatable), formerly unknown connections, or something otherwise deemed novel? I had to come to the realization many years ago that no matter how much time I spent reading and learning I would never consume even 1.0 x10^-10 of the information that exists that is currently written, or recorded in some other format. It was at that point that I realized that reading based on personal interests not only is enjoyable, but also allows for new social relationships to bloom, for the production of output (blogs, books, videos, conversations etc). One could argue what is relevant to x or y but ultimately relevant to x or y is seemingly a function of something you're either interested in or interested in finishing (i.e. a college class or whatever).
@zhanzh3ng said:
I think this question is another way of asking half of the original question of this discussion. My answer (which is not the only possible answer, obviously—@MartinBB gave the very different answer that "all information is valuable") is that I make an intuitive judgment about what will answer my questions or solve my problems or will question an existing answer or solution to my questions or problems.
I think of this in terms of defeasible reasoning, and you gave a good example of defeasibility above: I may at first think that "1+1 will always equal 2" is a good answer to a question in my note system, but then I learn that @zk_1000 overloaded the operator in the software I am using. So the answer is still relevant, but false: Now I have to find some way of indicating that "1+1 will usually equal 2" but not in the software application that @zk_1000 built for me. That's an example of defeasible truth. I presume relevance is defeasible too, but I can't think of a situation when I discovered later that information that I assumed was relevant was not. So I must be good at making my own relevance judgments.