Managing huge constant inflow of diverse information and possibilities to use it?
Hello!
Don't know if this problem belongs to Workflows or Research & Reading category
As title says, how to manage such inflow of info and (lingering) diverse bits of info (and use them)?
I was researching this forum but couldnt find the solution.
I will try to provide my description of problem and possible solutions, but i feel there should be more elegant way of doing it.
I'm aware of collector fallacy and low value of inactive ideas.
(I imply here that action may be also a relation)
I will name an item in the following text any atomic idea, observation, concept, relation, insight, fact, definition, tool, app, inspiration etc.
Even with filtering out the inflow the problems persist.
Situation:
Huge inflow of materials (from references, wandering around the web etc), observations, conversations, random thoughts etc creates huge amount of scattered, diverse and unrelated items.
- Some amount of items impossible to relate anywhere at the moment.
- So they stay in that big pile or amalgam inside inbox.
- You are sure that those items are important, you have come up with usage for them. But it's impossible because of missing parts you don't know of (lack of gathered information/knowledge).
- Some items create "projects" for further research/wandering, increasing possible amount of inflow even more.
- Some items demand action in the future, when more items will be gathered which will be able to connect to that item. But it's uknown which items you have to gather and from where.
- That leads to losing and forgetting them.
- Creating bottom-up structures also hard because of lack of required items to form structures
Having goal-oriented reading may lead to missing important items or relations.
How to solve it? Process everything to the bones, or mark that source as "partially processed"? (if you are sure that there's more than that you extracted for current goal)
Result:
- Lots of directions to research, but forgotten;
- Lots of scattered items in pile, demanding action but without possibilties for action because required items are not found or not created yet, or time hasnt come yet;
Examples:
- You accidentally got best fuel, but you can't use it without a car. Throwing it away is also unreasonable.
- You got an idea for machine learning. But you can't implement it before you gather more knowledge.
- You saved an article. You sure it will be very valuable once you will have more related knowledge. But it's unusable now. Furhtermore, you can't extract items because of lack of related knowledge.
- You got items which created directions for research or learning. You don't know if it will be even possible to connect those directions later into something new.
- Also hard to track profress on those directions.
- You got an article/post on zettelkasten. To fully understand/implement it, it requires some tacit knowledge you are not aware of. To make things worse, you gathered decent amount of such articles/posts. It's very hard to understand in which order you should process them and how to make good mental model out of them.
- You gathered some random items. They are certantly valuable, it will be hard to find them later outside your notes. But they become lost in your notes too, you can't use them now.
- Also any "possibly useful reference" which you gathered from (re)search - sources which you know may be useful, but they are not useful now. Spending time on them now seems costly. Simultaneously, you can't create new possibilities without extracting items for them. So you have to "take a risk" at processing them without any good result.
Possible solutions:
Create lists of "project/action" notes - containing directions, next actions for (re)search.
- Still hard to decide if it's worth it to dive deeper with that now, or should you choose smth else
Incremental Reading (separate, such as SuperMemo, or implemented inside ZK)
- For research, directions, gradual processing
- Every new possibility to dig further loads up into IR software . IR then handles all the gathering and processing
- Reading of folgezettels also can be implemented through IR
Spaced Repetition System (separate, such as Anki, or implemented inside ZK)
- For keeping items at the top of the mind
- Brain can't use what is not inside it's memory. Or almost the same - lingering items may be locked under myriads of associations layers, so it may be almost impossible to make a unconventional connection when scrolling through notes.
- Any item which needs to be kept easily accessible to your short-term memory
- Any item which can become actionable any moment (so you definetly will remember about it in such moments)
- Brief intro to SRS in comic form
Pick one itemand go through every other one-by-one
- For connecting items, forming topics bottom-up
- That gets VERY time consuming when amount of items grows
Use NLP, LLM, Vectorising notes
- For finding clusters, similar notes, "next" or semantically near notes
- That almost automates manual scrolling notes one-by-one. But what about very hidden, almost metaphorical connections?
- Also may be useful for clustering sources/materials/references, but it requires having full copy saved
- Obsidian with Smart Connections plugin is able do it with local models.
Side solution for developing inklings, connections between notes, creating manuscripts
- Relies on usage of SRS and IR, called Incremental writing.
- More on IW here - Incremental_learning#Incremental_writing__
Howdy, Stranger!
Comments
Never thought I would say this, but for you, I would HIGHLY recommend reading Building a Second Brain.
You need to develop a single question to think about as you process each item. This reminds me of Marie Kondo's best piece of advice in the book The Life-Changing Magic of Tidying Up. As you wonder to yourself, " How will I deal with all of this stuff? , she recommends you pick up each item one at a time and ask yourself, "Does this spark joy?" If the answer is no, then discard the item if you can or store it somewhere else (somewhere far away).
So, you need a question like, " Will I be writing about this or will it help me be a better thinker?", if the answer is yes, then process the hell out of it and put it in your Zettelkasten. If the answer is no or you are unsure, put in figuratively in one big container marked "Hold" or "Archive" and leave it in there until you decide one day that it is now worthy of writing or thinking about.
Read about it 2 days ago, and article about marrying ZK and BASB in particular
In BASB terms, such items would sit in Archive or Resources, as i understand it.
In my case it started breaking when the volume or of "unguided search" becomes very mixed/heterogenous
Yes to the latter most of the time, that's why lots of items become produced - they will help think better, but...
So as i see, pseudo-SRS approach is better applied here (what i do now) - items loaded up into SRS, and they are scheduled to show up with increasing intervals.
Items which are unusable for a very long time can grow intervals more than a year, which almost equals to "Archived for good" (and can be checked manually later, which is also equals to visiting archive manually).
That also removes the burden of manually looking through items, gives the effect of interleaving/serendipity, and focus.
You can easily go through 300 items daily in 10-20 minutes that way, if you spend around 5 seconds per item (to decide do you want to take action on it today or skip it).
That also may be called as "spaced inbox".
But due to our changing levels of energy during the day and week, there should be some sort of sorting by your context (tired/aware/at home/on mobile/etc) or you accidentally may throw items to very long intervals.
I overcome this by delaying for 3-7 days instead of skipping with interval increase (cases like "i would interact with it when i have more energy")
I meet, everyday, much more resources and ideas than I can handle, I think that this is normal in our digital lifes.
My solution, avoid the anxiety of having to manage them all.
I've a note in Obsidian, "interesting things", it contains hundreds of links. I will never process almost all of them, It does not matter.
Some of the links sometimes makes an aggregate that has best chance to spark my interest in a day, so they form their own note.
In a very bottom up way.
One day, in the future, using searches or following links between aggregate notes something will resurface.
During these months for example I sometimes met an idea or a resource regarding the development of an AI based chatbot.
Interesting, but now I don't have time and will to develop the idea, even more so the entire project.
So, I have a "my chatbot project" note and I throw all in there, in the raw/semiprocessed state that I can afford at the moment.
I remember now nothing about what I've collected in the past regarding chatbots, but I remember now that I have a chatbot raw note.
If and when the day that I have time and will to develop that project arrives, I will simply retrieve that note, "rediscovering" all.
Not all the stuff that enters into my system need to be processed. For much of them I only give a chance to resurface in the future, thanks to only a small bit of structure or a simple search.
I don't care of a chance loss, even losing an interesting chance I've more things to do than I really can.
thanks for sharing the Spaced Repetition comic, I didn't know that. It's beautiful, and this illustration even fits the Zettelkasten practice
Author at Zettelkasten.de • https://christiantietze.de/
@ctietze Great images - thanks!
The previous discussion "How to handle ideas for further exploration" (December 2023) may be related.
@Ydkd said:
Also look at David Allen's Getting Things Done (GTD), which is one of the precursors and foundations for Tiago Forte's Building a Second Brain. These days most knowledge workers seem to be familiar with GTD, but I don't see any GTD terms in @Ydkd's original post except inbox (which is not unique to GTD and may not indicate GTD knowledge). GTD categories such as someday/maybe, waiting for, and contexts could help if you're not already using them or a variant of them.
Also, in Ofer Bergman and Steve Whittaker's The Science of Managing Our Digital Stuff, there are three main principles: organizing by project, by importance, and by context: internal context (your thoughts while interacting with an item), external context, social context, and temporal context. Even when I save items just for reference (GTD) or resources (BASB), I try to maximize the contextual cues that I save with it, so that when I search for items later the appropriate contextual connections will be revealed. For example, if I'm just saving a file for reference or bookmarking a link without writing a separate note about it, I will still add relevant tags to it and write about what I expect the file to be useful for or what I think about it (internal context) in the filename or annotation field.
All of this is less about having a particular software solution and more about specifying a management system that works for you using concepts like those found in GTD, BASB, etc.
@andang76 said:
Since @Ydkd seems to be very interested in spaced repetition, I would point out that every time you search for something or revisit a saved search or review some other list of items, that action can be a kind of retrieval practice if you're really thinking about what you're looking at. Of course, such actions are not regularly "spaced" unless you schedule them, but since I am most likely to search for something when I have forgotten something about it, searching seems to me to work quite well for refreshing my memory.
EDIT: Also, the weekly review in GTD is, in part, a scheduled time to review important lists. The same is true for less frequent scheduled reviews, e.g. quarterly review.
@Andy
GTD categories such as someday/maybe, waiting for, and contexts - i have them. Someday/maybe/waiting-for almost the same in my case, so they all sit in the inbox or have "pending" status.
"Someday" almost always equals "never" from my experience and what i saw from other people
what do you mean by that or tell more about how you use it please.
I don't use pure spaced repetition for inbox. I use spacing part and queue part for managing lingering/pending items, the recall part is irrelevant here. Items also may have fixed interval (or capped - e.g. interval will grow until it reaches 27 days). for review. SRS is being used here just for passive or semi-active review of items - you can skip it or take action (think about it, trash it, rewrite it, elaborate etc.).
It removes context switching and focuses your attention on pending items, because going through links and searches may be unreliable.
from what i found out, it more gives sense of familiarity and now improving memory itself (very widespread across students, where they read a book and feel like they know it, but after book is closed they cant remember almost anything)
Also searching is more time-consuming (manual search + sorting out not pending items)
Andy matuschak also wrote on this:
@Ydkd said:
If that works for you, OK, but that's very different from standard GTD, where inboxes are separate from the lists where you put actionable items. You may want to explore changing your current system so that it's more like standard GTD; it may help.
Yes. It's an inventory of things I'm not getting done! But I differentiate active someday/maybes, which I'm not doing yet, from archived someday/maybes, which I've decided not to do. They are separate lists.
I used the word context as it is used in The Science of Managing Our Digital Stuff, where it refers to circumstances and conditions in which information is created, organized, and used. When I said "I try to maximize the contextual cues that I save with it", I meant I try to record information about the context, because context is not necessarily automatically recorded. Temporal context in the form of a timestamp is automatically created when I create a file. But the project, the importance, the internal and external contexts, and the social context are often not automatically recorded, and I have to record that kind of information myself. The context of use doesn't exist yet, so I have to imagine what it is and record it: In which circumstances and conditions am I going to use this item?
Thanks for clarifying that, I didn't infer from what you wrote that the recall part was irrelevant. So it sounds like you are doing a more quantitatively sophisticated version of GTD reviews of items.
I'm not primarily trying to improve my memory (and it seems that you aren't either, since you said "the recall part is irrelevant here"?), but the reviewing that I'm talking about is different from "ineffective" memory techniques like rereading and highlighting, which tend to be only familiarity/recognition tasks, as you said. The reviewing should be more like self-testing. When I review, I'm asking myself questions such as "Is this still accurate based on what I have learned since I last reviewed this?" and "How does this relate to I have learned and created since I last reviewed this?" Also, new searching is already more demanding in this regard than merely rereading, because you have to generate appropriate search terms, and to do that you have to quiz yourself about the context of what you are searching for.
It is difficult to say that this is true or false in general; I imagine that it depends on how one's system is designed and the purpose of the reviewing. In your original post, your list of kinds of items and your list of problems are heterogeneous. I imagine that there will be different solutions for different kinds of items and problems.
Here's an excerpt from the summary of Part I of The Science of Managing Our Digital Stuff:
I added emphasis to the phrase "organizing actionable information remains a major challenge". The more information you try to manage, the more challenging it will be. So, of course, "managing huge constant inflow of diverse information and possibilities to use it" will be challenging.
I've read some of what Matuschak wrote before, but I'm going to revisit it and see if I can use more of his insights, so thanks for sharing that. He has many good ideas, but sometimes he underestimates the value of traditional knowledge-work methods: see, e.g., Josh Bernoff's "Why books work: a rebuttal to Andy Matuschak" (2019).
@Andy
thanks for the replies
do you have more such good resources on knowledge work (even unrelated to notetaking per se)?
besides those mentioned here:
https://forum.zettelkasten.de/discussion/2938/whats-your-list-of-three-books-to-start-with-zettelkasten
https://forum.zettelkasten.de/discussion/2824/books-about-personal-knowledge-management-and-zettelkasten/
@Ydkd: One thing that's missing from your original post is an explanation of how such "inflow of diverse information and possibilities to use it" relates to your own vision and goals for your life, or what David Allen calls the "higher-altitude" perspective in GTD. Perhaps you know that but chose not to write about it. If you feel the need to think more about that relation, you may benefit from looking at Allen's book Making It All Work: Winning at the Game of Work and the Business of Life (especially the later chapters on "Getting Perspective"), which was one of the sequels to Getting Things Done. There are other books by other authors on the same theme, which is not specific to knowledge work but is more generally about what Allen calls integrated total life management.
It sounds like there is an imbalance between your information diet and your actual knowledge work. I'd start by cleaning that up.
Being hyper-exposed to "useful, but not useful now" is a major symptom of not attending to the relevancy aspect of your information diet. All that time exposing yourself to information that is not actionable should be re-allocated towards actual processing and putting into action.
For the rest, this is my approach: https://zettelkasten.de/posts/building-a-second-brain-and-zettelkasten/
I am a Zettler
@Andy
The vague goal is to be able to do whatever knowledge/creative work with minimum time spent on system actions (search, maintenance, being stuck etc), and maximum time spent on actual thinking, creating etc.
I gather lots materials and observations about cognition, science, tech, learning, various princliples etc. I have several big results emerged of such "wandering across information without any specific goal", but i am not satisfied with the result/time ratio.
Also not satisfied with (my) lack of tools/principles on fast exploring/researching topics without clear borders.
On inflow:
we are (or our actions at least) ultimately limited by our knowledge, and there's no way other than to use abundance of information to increase possibilites to improve our life and decide what you can even do.
On more general scale - i want to develop a ultimate system and tools for thought for almost all types of knowledge/creative work (both software and principles) for everyone from artist to scientist, saving as much time and sanity as possible.
That's one of my very vague goals/directions.
As an example, spaced repetition system (with complex algos) solves memory part of knowledge work - you can open it at any time, and ratio result/(time*effort) will be almost maximum possible (depending on card preparation), no empty time on search, remembering what to review, trying to decide what exactly you don't know/remember. It's efficiency is almost maximum at any moment, be it usage for 5 minutes or for an hour.
Tools like tana, logseq, notion, anki, obsidian etc shaped my paradigm to the point that i cant go back to traditional Pen&Paper or MSWord&GDocs for everything, i feel like i almost lose my limbs.
So i wonder what such ultimate system & principles for any knowledge&creative work should be like?
@Sascha
Read you article on BASB+ZK already, i'm using something like this already but more automated.
Already limited my time doing this
thats one of the major parts - impossibility to define relevance of certain items without processing them
Now i'm also trying to find best tools on semantic searches, clustering, automatically relating information and other things
In obsidian there's smart connections plugin, but it also needs additional tweaking
Here are test vault with 15 separate notes. There are no explicit links [[like that]] between any two notes, there's no info about content in their numberic filenames.
To the left is currently opened note, to the right notes related by Smart Connections plugin with local embedding model.
Here are one single note, to the right - embedded blocks inside single note, related to the highlighted text inside the same note.
But that don't work quite well so i'm still in search of other tools and approaches
i gathered some apps, but some are limited, some are changing quickly, hard to orient and make use
some apps i gathered if anyone interested too:
reor (https://github.com/reorproject/reor);
github.com/sigoden/aichat/;
saner.ai;
raindrop.io
fabric.so
betterstacks;
iki.ai;
myreach;
gopeek;
zenfetch;
Seems like a too big scale, from my point of view.
First of all : what are you building right now? Are you trying to build your own PKM, or a tool for people that you will distribute later? I may have misunderstood your posts, but until now, you seem more interested by developping a tool right for you and you know what? It is a quest by itself.
Oh, I know it, it is called "human brain".
Tools are one thing, the way you use your brain's capacities is an other one. The "ultimate" solution would be the one to work in symbiose with the way your brain works, in the simpliest and smoothest way possible.
@Ydkd said:
What you just described sounds to me like processes (means) instead of goals (ends). Improving processes is a good goal to have, but perhaps it should be subordinate to other goals so that you can judge how much process improvement is needed to achieve the other goals? You may be able to get more thinking done if you focus on other goals instead of focusing so much on process improvement.
I wouldn't discourage anyone from gathering as much information as they want, as long as it's not impeding them from attaining their goals. But your description of your approach reminds me of a quote from Francis Wade about "boiling the ocean" that I have shared before:
As Wade and others have said (for example, here and here), the opposite of starting with information and trying to boil the ocean is instead starting with the description of problems and goals, and deciding on solution strategies for specific problems.
By the time we have matured enough to be able to write as coherently as we do in this forum, we probably already have enough knowledge to start with problems instead of with information. Or as @Loni said about the "ultimate system & principles for any knowledge & creative work":
Good one!
i am interested in both for me and for other people
it's a general direction, there's much to solve for sure
thankfully memory part is solved already
tools still shape brains capacity. Our memory is nearly unlimited, but without SRS its very hard to use it to the fullest. SRS is one of such tools for thought that empowers our cognition and creativity with hight efficiency (but still needs guidance to reach that, such as Wozniak's 20 rules of knowledge formulation).
Our brain also needs guidance.
has to be able to be as simple as needed and as complex as needed. It is one of alredy existing such tools.
Tools also may be formed as principles, practices and not "physical" tools. Zettelkasten is such tool.
Hyperlinks are also tool for thought.
Abstractions like "file", "document", "database", "graph" can be separate tool and part of bigger tool for thought. They not only empower thinking&creativity, but shape it.
Overall, such tools for thought should be based on cognition, and should be possible to bend to user's preferences - like Zettelkasten has basic principles as a system but you can modify it to your own workflow. Or tana/logseq/obsidian/archive/windows filesystem - they provide both their own workflow and flexible building blocks to shape them according to your will
Again, thats a very vague direction
Doing it already (i just dont list them here, because they are not related specifically to this thread), that's why need for such systems emegred
Obviously im very unlikely to create such system, and more likely i will create single solutions per 1-2 types work (which hopefully could be extended to other types), like SRS, and it will be faster to come up with ideas in fast iterative cycles
yes, i'm trying to develop workflow which will allow to spend very minimal time on this while getting results (whatever needed)
Yep; I agree with that concept. I think of it as "minimizing friction" at all levels and entry points.
@Ydkd: Another thing that came to mind is environmental scanning in business strategic planning/management. Environmental scanning is mostly about monitoring a changing strategic environment. It seems relevant because much of your information-seeking behavior described in the original post seems similar to scanning.
See, for example, this article: Chun Wei Choo (2003). "The art of scanning the environment". In Voros, Joseph (ed.), Reframing Environmental Scanning: A Reader on the Art of Scanning the Environment (pp. 7–18). Hawthorn, Vic: Australian Foresight Institute, Swinburne University of Technology.
See especially the figures in the article. The article is not about information system implementation, but summarizes a simple conceptual framework. The framework may provide a different way to think about your relationship to information seeking. Chun Wei Choo's modes of scanning are: undirected viewing, conditioned viewing, informal search, and formal search. Each mode corresponds to a different information need.
At one point in the article, Choo said:
I'm reminded of the quote above from The Science of Managing Our Digital Stuff that said, "organizing actionable information remains a major challenge." What Choo said seems to corroborate that: successful environmental scanning for actionable information is a major challenge that requires "a critical mass of talent and resources" and can take years to develop the relevant knowledge and expertise.
@Andy: big thanks, i'll read through everything you mentioned