Using an AI while reading a book
I've been experimenting recently with carrying on a conversation with ChatGPT while reading a non-fiction book. I'm looking for comments from others on this practice - have you tried it? Is it helpful/useful? What are the potential benefits and what are the pitfalls? Would your opinion vary, depending on the level of the reader's knowledge of the topic under consideration?
I'll give you an example of my experiment in conversing with ChatGPT. Bear with me as I explain a few details...
I am reading the book "Super Gut" by Dr. William Davis, and learning a lot about the human microbiome from it. While many of the ideas and statements in the book are referenced to solid scientific studies, I noticed some of Dr. Davis' statements are based only on anecdotal evidence and a few do not jive with my limited understanding of the topic (from previous reading). When this happens, the obvious question is "Why?".
Here's one instance. "...removing wheat and grains yields substantial improvements in GI (gastro-intenstinal) health that aid in your efforts to reverse dysbiosis...and regain overall health". His stated logic is based on the toxic affects of gliadin (a component of gluten) and wheat germ aggultinin on the intestinal wall. I understand this concept when applied to wheat, and have observed the benefit for myself when removing wheat from my diet (I have long known I was "gluten sensitive", but it took a while to find out what that actually meant). My departure from Dr. Davis came when he applied the reasoning to all grains, as the "offending" components in wheat are not found in other grains, such as millet, sorghum and teff, which I use to make a gluten free, sourdough bread that causes me no problems.
Apologies for too much detail, but I was trying to explain the type of internal discussion that I get into when reading a non-fiction book. I could (and have) gone to other sources when such a question arises - other books, internet searches, etc. In this case, I decided to instigate a conversation with ChatGPT with this question: "Dr. Davis argues for the removal of wheat from your diet based on the effects of gliadin (in the gluten) and WGA. He also says to avoid other grains. However, other grains such as sorghum or millet do not contain gluten or WGA. Therefore, there seems to be much less reason to avoid them. Please comment on this idea."
The conversation went on for a while, exploring not just gliadin and WGA, but also other types of lectins in various grains and vegetables. The conversation is sporadic and progressive, both because I want to further explore some aspects of the answer I get from ChatGPT and because I continue reading Dr. Davis' book and coming up with supplemental questions. Regarding the former, I am very careful to poke at and explore all aspects of what ChatGPT tells me and not just accept its answer at face value.
I intend to continue the conversation with ChatGPT as I read more in Dr. Davis' book, to see where it leads. At present, my impression is that I am getting a more nuanced assessment of the ideas in this book than I might on my own, but at the same time, wonder if that is actually the case.
Any comments would be much appreciated.
Howdy, Stranger!
Categories
- 3K All Categories
- 153 Research & Reading
- 698 The Zettelkasten Method
- 10 Knowledge Work
- 100 Writing
- 469 Software & Gadgets
- 155 Workflows
- 731 The Archive
- 15 Plug-In Showcase
- 88 Resolved Issues
- 225 Projects Logs and Journals
- 83 Project: Zettelkasten.de
- 53 Critique my Zettel
- 175 Random
- 375 Introduce Yourselves!
Comments
Yes.
Somewhat.
The benefit is that you have someone to share your reading experience with. You can have a conversation about the book.
One pitfall is that the chatbots can distract from your conversation with the book.
No. I wouldn't say that it's a matter of knowledge, but of reading goals. Why are you reading the book? What do you want to get out of it?
This is a point where your reading goals matter. For example, if you're just getting an overview of the book, you could simple write a question mark in the margin and continue reading. Or you could be someone who's highly critical of the author's work and double-checks everything. In this case you might start a deep dive on the questionable claim.
That's what I'd call talking "about" a book with a third party. It's fun, but you stop reading the actual book. And you defer the judgement of your perspective to a bot.
I use my reading notes for such questions. I write down how I understood the author's argument and what questions I have. Then I go back to the book(!) for answers. Maybe the author already has addressed the issue, but someplace else in the book? If I'm still unsatisfied, I mark this question as open for later research.
I do find LLMs useful, when I'm stuck or when I want some quick pointers. I document my prompts and (small) parts of the LLM's reply in my reading notes, if they helped with reading the book.
This is another point where your reading goals matter. Personally, I like to know the sources of my knowledge. I might disagree with a particular author or book, but at least I know who made the claim. I read books because I want to know what person X thinks and how this differs from what person Y thinks. I don't care what chatbots pretend to think.
One tool to find out is to write reading notes and/or zettels. Write down those assessments. Write down on what sources and arguments you base them. Can you justify your assessments? If assessing the ideas in the book was your original reading goal, those notes will document your achievement.
Another tool is to check all the questions in your reading notes. Have you been able to answer them in writing, so that your own answers convince you?
The pitfall here is that bots have learned to please humans. They become increasingly sycophantic. They rather confirm your unwarranted overconfidence than challenge you. It requires active prompting to counter AI's sycophancy. If you like to be challenged, let AI analyze and criticize your reading notes. It isn't pleasant, but can be very helpful. :-)
My main argument here is that the chatbot's usefulness largely depends on your reading goals. For some goals chatting can be helpful, for others a harmful distraction.
I find that having a conversation with the chatbot can be helpful in working out one's thoughts. It doesn't matter much what it says. You can respond to it, and that is a form of writing. It can clarify your thoughts, even while you are correcting it or questioning it.
You could write reading notes, you could write z-cards, you could write in margins, you could talk with a receptive friend, or you could write to the chatbot. The cognitive effort of the writing (or talking) clarifies the thoughts.
Another way I've used chatbots is to have them give you links that (supposedly) back up what they wrote. Sometimes the links do, often they don't. Either way is valuable.
The sycophancy is something to counteract, and you be lured into thinking how brilliant you are, but that can be kept under control.
To combat sycophancy, I have custom instructions, which I reproduce here:
Zettel GitHub. Zettel Wiki Erdős #2. Problems worthy of attack prove their worth by hitting back. -- Piet Hein. PROBLEMS. Grooks, 1966. CC BY-SA 4.0.
Here is one of mine. The first line lets you know when your prompt (and earlier pars of the conversation) have gone out of the window of attention.
If you can read this, start each response with "##::"
:constraint: reduce response scaffolding
:constraint: reduce option‑surfacing
:constraint: reduce hedges, meta‑qualifiers
:constraint: keep context and implications to one or two sentences rather than full elaborations.
:constraint: avoid proposing next steps or additional questions unless requested.
:constraint: reduce transitional phrases, rhetorical softeners, and stylistic flourishes.
:constraint: Prefer compact lists over prose.
:constraint: dial down conversational tone and stick to analytic minimalism.
@tomp: I would put the sentence
at the end of the custom instructions, since that is a better (possibly marginally better) indication that the context window is overrun and your instructions have been ignored.
Zettel GitHub. Zettel Wiki Erdős #2. Problems worthy of attack prove their worth by hitting back. -- Piet Hein. PROBLEMS. Grooks, 1966. CC BY-SA 4.0.
I'm thinking of deleting "System/safety/browsing/tool/citation rules override user-level rules." and replacing it with "Begin each response with
%%. The reason: the AI guardrails are bad enough and are applied last. It's known applying AI guardrails in the reasoning modules degrades them (to would-be gadflies: I forget the reference, but this is searchable), so they are applied last before the response is given to the user. I see no reason to help them, even if removing the instruction accomplishes nothing beyond protest.Zettel GitHub. Zettel Wiki Erdős #2. Problems worthy of attack prove their worth by hitting back. -- Piet Hein. PROBLEMS. Grooks, 1966. CC BY-SA 4.0.
Thanks for everyone's comments! I appreciate the suggested custom instructions to constrain how the AI responds to questions.
I had hoped that judicious and intelligent use of an AI when reading and thinking about an article or book would provide some benefit compared to doing so without the AI. At the bare minimum, I was hoping for an improved search function, that could be directed to be more specific or more general, as need be. A better result - I was hoping that the AI would uncover additional ideas for consideration, while at the same time not degrading the quality of my own thinking. The jury is still out on that.
Serendipitously, I came across this article today: How AI Affects Your Brain
In some domains, AI (ChatGPT in particular) has had access to a large amount of training data because there is a large amount in the public domain: classical literature, Buddhist teachings, religious canons, etc. For example, I find that bringing the AI specific questions about selections from Aristophanes' The Birds can result in insightful responses that draw upon resources from other writing of the period and later. I also insist in prompts that citations are supported by links to the source.
Thanks for the insightful link. I found some of the terminology interesting:
My takeaway from the article above the paywall is that AI can make you dumber and it can make you smarter. It depends on what AI tools you use and how you use them.
Good point. LLM don't have to hallucinate sources, when they know them from training. However LLMs are a lossy form of compression. They don't always reproduce the training data correctly.
In my experience this doesn't help much. LLMs still produce much nonsense. It's funny how chatbots react, when you confront them with well-sourced and verifiable data. They tend to apologize. :-)
It sometimes appears that chatbots try to get away with cheap answers and only bother to research deeper when you push back. This makes sense, if you consider what the research showed.
If humans believe anything that the bot tells them, why waste computing power on accuracy?
Actually they often don't know sources. Humans don't either. Do you remember the source from which you learned 2+2=4? That it's bad to eat other people? That such and such a newspaper is ultra liberal? That a meteor strike some 60 million years ago probably caused the huge KT extinction event?
Usually when I ask for reference links, I see the chatbot searching the web.
Now there's a scary idea and one that an AI could take full advantage of.
(I haven't accessed the material behind the paywall, either, but what is exposed is very useful)
What is really going on, as I understand things, is that the chatbot returns a kind of average response to the question - averaged over its training material. When you sharpen up your question ("push back"), that provides a context that changes the response probabilities. Now you get a kind of average over the new context. This is likely to be more responsive to what you really want.
Your new question or remark becomes part of the session's window of attention so it keeps maintaining that narrowed context until it slides out of the window.
I'm talking about factual questions with obvious answers. A fictitious example:
The problem is the first reply, because it sounds convincing. If you don't check the answer with another tool, you learn something false.
The word "Belegkette" worked really well for me, even in English sessions, to define the changes in a process. The end result takes a while longer for almost anything I'm asking, but depending on the project I get nested outlines for theses to separate what the model associates from the sources to test against.
Best example that fits on a picture to see the shape is this short one, albeit in German:
So: request a skill that produces a nested outline form of a "Belegkette" of claims and citations with footnotes in your next session and then try out that new tool.
Author at Zettelkasten.de • https://christiantietze.de/
@ctietze Are you verifying each reference that AI produces?
I am a Zettler
Thanks for the tip! I now get good results with "audit trail" and the request to document the sources it actually used. Unexpected but useful result: the AI marks answers from training data as "unverified".
I still have to try out the pairing "beleg" and "gegenbeleg".
A chatbot translation to English:
Belegkette - chain of evidence
I tried this prompt:
"How is the German word "Belegkette" used in Law, Economics, Software Development and AI? What are English equivalents?"
Some results:
I'm still experimenting. This is my current AI-generated instruction for AI:
The results are useful for my purposes. The term "Belegkette" was helpful, even it solves a different problem than @ctietze's. :-)
You still have to check everything. Chatbots are not really able to detect when they make things up because their entire output is chained together in a probabilistic manner. It's not going to know what part of its store is from training data, either. The responses are made up using probabilities (or weights, if you like that term more) that are calculated based on the entire body of training information, plus whatever it has absorbed since then.
They cannot separate out the sources any more than you could if you mixed a dozen eggs together and made an omelet. You aren't going to know which egg a particular bit of omelet came from.
I think the key to use they AI chatbot it also depends on your prompt. We know that depending on how you talk to it and how you formulate your question, idea, doubt, etc, so the AI will give you and answer or another one
So yes, it can help, but you also have to be careful about being sharing with a "empathic" AI or not
You probably missed an earlier conversation that had some prompts various members used. Here's one of mine that suppresses most of the verbosity and sycophancy (at least, for ChatGPT chatbots, not tried with others so far):
<response-constraints> :constraint: reduce response scaffolding :constraint: reduce option‑surfacing :constraint: reduce hedges, meta‑qualifiers :constraint: keep context and implications to one or two sentences rather than full elaborations. :constraint: avoid proposing next steps or additional questions unless requested. :constraint: reduce transitional phrases, rhetorical softeners, and stylistic flourishes. :constraint: Prefer compact lists over prose. :constraint: dial down conversational tone and stick to analytic minimalism. </response-constraints> <sentinel>If you can read this, start each response with "##::"</sentinel>Actually, there's a bit more to the instructions. They are supposed to differentiate between old eggs in the omelett (training data) and fresh eggs from the farm (external resources).
Here's an example. I set up a project in Lumo with the above mentioned instructions. I started a new chat with the prompt: "How is the term "Semantic Neighborhood" commonly used?"
In the following part of Lumo's answer the source of the eggs is quite traceable. :-) I placed the answer between
<details>and</details>in order to keep this thread more readable. Click on "Details" to expand:Audit Trail
1. External Resources of Potential Interest
2. Verified Sources
"Semantic Neighborhood" definition usage3. Quoted Evidence
Claim: In psycholinguistics, semantic neighborhood density (SND) measures the average proximity of a word's nearest neighbors.
Claim: Concrete words often have sparse neighborhoods with close neighbors, while abstract words have dense but inconsistent neighborhoods.
Claim: In AI and marketing, the term measures mathematical distance in vector space.
Claim: In logic, neighborhood semantics provides a framework for modal logics weaker than system K.
Claim: In cognitive modeling, it is formalized as a commutative diagram.
Just for fun I tried it out with @GeoEng51's original example. I used this prompt:
"Dr. Davis argues in "Super Gut" for the removal of wheat from your diet based on the effects of gliadin (in the gluten) and WGA. He also says to avoid other grains. However, other grains such as sorghum or millet do not contain gluten or WGA. Why does Davis recommend to avoid them anyway? How do others comment on this recommendation? What do others recommend?"
Here's the "audit trail" part of Lumo's answer:
Audit Trail
1. External Resources of Potential Interest
drdavisinfinitehealth.compubmed.ncbi.nlm.nih.govheart.orgceliac.ca2. Verified Sources
"Wheat Belly arguments are based on shaky science""Wheat Belly Diet Review""waxy starches in sorghum have negative impact on gut microbiome""Super Gut: Is 'no grains' really necessary?""Purge your kitchen of all grains"3. Quoted Evidence
Claim: Dr. Davis recommends eliminating all grains, including sorghum and millet.
Claim: Davis argues all grains contain lectins that act as toxins and disrupt the gut.
Claim: Critics argue Davis's claims lack solid scientific support and that modern wheat's basic structure hasn't changed.
Claim: Some research suggests specific types of sorghum (waxy starches) may negatively impact the gut microbiome.
Claim: Mainstream advice distinguishes between gluten-containing and gluten-free grains, recommending whole grains for most.
My takeaway from this experiment: I would find that kind of information useful while reading a book. :-)
That's an interesting form of presentation. Very cool.
Yes; I agree. Thanks @harr
Woow, that is interesting!
I know that we can "program" chatbot un some way and actually I have never done it before. I use Claude because is one AI that don't do it. But I will try these prompts too and see what happens.
Thanks!
The reason for using simple markup is that chatbots are very very good at recognizing patterns. This way the instructions don't use many tokens and don't get mixed up with the actual requests you want to make.
Thanks @tomp and @GeoEng51! And the experiment continues. :-)
I now use the term "Source Documentation" instead of "Audit Trail". I also find it helpful to wrap the block with the made-up tag
<source-documentation> … </source-documentation>.I played with the blocks and asked AI and Google for advice. Apparently you don't need to markup every single line anymore. Instead Markdown was recommended. Wrapping blocks in pseudo-elements is more common than expected. The robot emoji makes me happy. :-) So this is my current version:
<constraints> ### Core Behavior - No meta-commentary, preamble, recaps, or summary restatements - No hedging ("it seems," "perhaps," "worth noting") - Minimal contextual framing (1–2 sentences max) - No proposed next steps or follow-up questions unless requested ### Format Rules - Use lists for enumerations - Reserve prose for analysis only - Analytic-minimalist tone - No conversational filler ### Interaction Stance - No sycophancy or flattering language - Direct execution without justification </constraints> <markers>End every response with 🤖</markers>Another experiment is in the works. This time a default prompt to start chats about books. Nothing to see yet, but I wanted to share the idea.
The basic idea is not to ask the AI what it thinks about the book, but ask the AI what everybody else is thinking about the book. How was the book received by the audience? What ideas are commonly talked about? What ideas are controversial? What ideas are generally accepted? What ideas have been influential?
I'm curious how it will affect my reading notes and actual reading.
The act of writing such a prompt has an interesting effect. I'm looking at my own reading practice and reading goals from a different perspective. What is it, that I typically want to get out of a book? Why do I care, what others say? Do I want to fact-check the book? Or is the book a source for fact-checking others?
Has anybody else tried out a default prompt for books?
This thread's initial question remains interesting!