Proximity Search in The Archive surfaces rich link candidates when creating new notes
I want to share a method of searching for candidates to link ideas in a new note to established ideas that consistently produce high-quality results. This advanced search option is called proximity search. Usually, proximity search has to be implemented at the software level. My main goal is to share this idea with you.
I love making lists, so you'll have to suffer my indulgence.
- I use The Archive to host my ZK.
- I have 3K+ zettel, but you'll find this tool of use if you have more than a couple hundred.
- My notes are all plain text formatted with Markdown.
- I use and recommend Keyboard Maestro integrated with python and zsh scripts.
- There are other ways to accomplish the same outcomes using other macro and scripting tools.
I'm dividing this demo into three parts.
1. A few low-level and high-level ideas about connecting zettel. Mostly I assume you understand the importance of a deeply connected ZK.
2. A deep dive looking at a new tool I developed for searching large mature ZKs with large idea schemas. I stumble making the video which turns out low-brow. My thought processes are all over the place. Turn on the microphone, and I become an idiot.
3. A brief look at the technical aspects of this proximity search tool.
What are we trying to accomplish?
We want to populate our notes with deep, meaningful, high-quality connections. This gets harder and more time-consuming as the ZK grows. The keystone step in knowledge growth is finding relationships between new and established ideas.
The primary tool we use is "search" when we look for link candidates. We search for the occurrence of a target word or phrase based on where we are (on which zettel we are proposing a link from and back to.) In practice, this type of searching can lead to a large percentage of false positives. False because the ideas are not relevant and positive because each note contains the target word or phrase.
The tool in the demo is super helpful in narrowing the candidates to those with the highest quality and most relevance. It's sloppy to connect zettel with weakly relevant links. It can be time-consuming to sort through too many connection candidates.
Full text and Boolean-derived searches are less effective in a ZK of a few hundred notes in the same schema. There are caveats to this, Notes in schemas or ideas streams that use technical jargon are easily found with a full-text search; if we have a developed ZK where the same jargon is used in a hundred or more zettel, we may still struggle to get to the best link candidate with a full-text search.
The technical details in part three will not make much sense without viewing the video.
This is me wrestling in real-time, sometimes struggling to make sense, stumbling with vocabulary, and making a mess of my typing. I'm taking a new idea captured from a forum post and looking to see where I can connect it to established ideas to strengthen it, question it, advocate for or against it, and reconcile it. I do a passable job giving a play-by-play.
Some or none of this will not directly apply to you. You will have to see where things might use in your world. Take what is helpful and ignore the rest.
We use a note inspired by @JoshA's Object Level and Meta Level Abstraction Distinction Applied to ZK forum post.
Object Level and Meta Level Abstraction Distinction []
Here is the original note before we integrate it into the ZK.
https://www.dropbox.com/s/x5k8npljh3q9dpg/Object Level and Meta Level Abstraction Distinction (Before) 202208190944.md?dl=1
Here is the note as it is today.
https://www.dropbox.com/s/l4bkxggm0lqbwyk/Object Level and Meta Level Abstraction Distinction (After) 202208190944.md?dl=1
- I'm sharing this with the hope that you will get some ideas on implementing the best parts of this in your workflow.
- It would be interesting to prioritize those with a closer distance score. Of course, we'd have to get a "distance score."
- This Keyboard Maestro macro is not super advanced.
- Each zettel must have a UUID (Universal Unique ID).
- The way I've implemented it requires:
- Keyboard Maestro
- zsh and egrep
- There needs to be a UUID for each note
- For those who use The Archive, we may have to wait and see how @ctieze implements scripting.
7. Here is the Near Search Keyboard Maestro Macro.
Ideas for future enhancements -
1. We'd have to get a "distance score," which might be converted into a "relevance score."
2. I'd like to rank the hit candidates by some criteria.
3. I'd like to have a way to eliminate the obvious false positives from the list.
4. Be able to choose the highlight color for ALL the search terms.
5. Rewrite in python or have this be a possibility of The Archive's promised forthcoming internal scripting.
6. Be able to cycle through ALL the search terms with a keyboard shortcut.
7. If the search target is found in the title of a link, only show it once rather than everywhere the link is placed.
8. The ability to add an optional third proximity argument.
9. Use the near-term search in combination with Boolean language.
10. Key, I'd like to be able to include phrases.
The quality of our thinking is directly proportional to the quality of our reading. To think better, we must read better. - Rohan
It looks like you're new here. If you want to get involved, click one of these buttons!