Idea for a script to make structured link lists easier to browse

ctietze

1. Open a Zettel, e.g. a structure note (or any other list of links)
2. Run the script (missing in the workflow!) to extract all links, i.e. all the IDs
3. Convert the list of IDs into a search expression to produce a list of all (aka any) of the linked notes: OR them together
4. Search for this combined search term

Why is this relevant?

I want to browse all notes about a topic; it's a long list of links, and I don't remember details, so I want to potentially read them all.

Starting with a structure note ...

... clicking the first link changes the search results; so I need to navigate back to get to the structure and then select the second link, and so on.

That sucks. So what would the suggestion look like?

Let's look at the example in more concrete form.

Structure note example

# 201707071018 Markdown syntax highlighting with the Cocoa Text System
##outline #textkit #markdown #nstextview #syntax-highlighting

## Highlighting hashtags and links

- [[201711291443]] Take 2 NSLayoutManager passes for edits and highlighting
- [[201707071017]] Attach URLs to attributed strings after parsing
- [[201707281031]] Optionale RegEx capture group in Markdown Emphase

## Highlighting algorithms

- [[201707311731]] Markdown syntax highlighting real-world optimizations
- [[201708011016]] MacDown uses peg-markdown: async, delegate based
- [[201712071426]] TextMate uses async highlighting
- [[201712290820]] Highlightr uses async JavaScript engine per paragraph
- [[201712021434]] Ulysses uses sync syntax highlighting
- [[201708021910]] Uli Küsterers uses `NSTextStorage.processEditing(_:)`
- [[201804131911]] Textastic for iOS uses custom TextMate parser
- [[201712051954]] Avoid storing syntax highlighting information in the `NSTextStorage` as logical edits that trigger `processEditing`
    - [[201711280929]] Use temporary attributes in `NSLayoutManager`
    - [[201712061552]] Apply syntax highlighting outside NSTextStorage to the text view
- [[201712290946]] MultiMarkdown for syntax highlighting

## Writing a Markdown parser

- [[201711090800]] Use String.UnicodeScalarView for fast parsing

## List indentation:

- [[201708041031]] Outline for a text view list indentation algorithm

## Themes

- [[201712131208]] Draw custom selection color for inactive NSTextViews

Fetching the IDs

A list of grepped IDs from this note:

201712051941
201706290826
201712021900
201707121506
201711280929
201712051954
201711291247
201712061524
201712061528
201712131210
201707281035
201901161119
201707121506
201708031445
201712060919
201712061528
201708031048
201708031447
201803071015
201707121506
201901041758
201707081109
201707121417
201707071018
201910191438

The result: the structure note mapped to the search results

Combined into a OR-combined search string for the set of notes that contains either of these IDs:

 201712051941 OR 201706290826 OR 201712021900 OR 201707121506 OR 201711280929 OR 201712051954 OR 201711291247 OR 201712061524 OR 201712061528 OR 201712131210 OR 201707281035 OR 201901161119 OR 201707121506 OR 201708031445 OR 201712060919 OR 201712061528 OR 201708031048 OR 201708031447 OR 201803071015 OR 201707121506 OR 201901041758 OR 201707081109 OR 201707121417 OR 201707071018 OR 201910191438 OR 

To ignore links to the IDs, I use a literal search phrase for headings with the ID, e.g. # 201803071015 instead of just 201803071015. The concatenated result:

"# 201712051941" OR "# 201706290826" OR "# 201712021900" OR "# 201707121506" OR "# 201711280929" OR "# 201712051954" OR "# 201711291247" OR "# 201712061524" OR "# 201712061528" OR "# 201712131210" OR "# 201707281035" OR "# 201901161119" OR "# 201707121506" OR "# 201708031445" OR "# 201712060919" OR "# 201712061528" OR "# 201708031048" OR "# 201708031447" OR "# 201803071015" OR "# 201707121506" OR "# 201901041758" OR "# 201707081109" OR "# 201707121417" OR "# 201707071018" OR "# 201910191438"

This is what the result of the search looks like; note the absence of the structure note itself, 201707071018:

Now I can browse the structure note "contents" directly, 1 level deep.

Maybe that's useful?

Maybe that's worth a script or KM macro?

ctietze

With grep -Eo you can use regular expressions and limit the output to the match; I use two greps to first find links between double square brackets, and then "zoom in" on the IDs itself, to produce the list of IDs from the original post:

"the content of the note would be here"   |   grep -Eo '\[\[([0-9]+)\]\]'   |   grep -Eo '[0-9]+'

Surround the list of Ids so that 202004091957 becomes "# 202004091957" OR, add:

| sed -E ';s/^([0-9]+)/"# \1" OR /g'

... then to join all lines together:

| sed -e :a -e '$!N; s/\n/ /; ta'

---

Combination of the above in a single command, split (\) onto multiple lines for readability:

"the content of the note would be here"  \
    |   grep -Eo '\[\[([0-9]+)\]\]'  \
    |   grep -Eo '[0-9]+'   \
    |   sed -E ';s/^([0-9]+)/"# \1" OR /g'   \
    |   sed -e :a -e '$!N; s/\n/ /; ta'

JonJia

Good idea!

To extend, I think that only need to add a "open in new Tab" to the current "ZID link", it can also solve this problem

discordian

Hi,

This is a good idea! I use similar thing a lot on my own system. It looks like this:

Video of my system

Since I live in the terminal, I can use fzf to preview the notes as I go.

I have other similar actions that select from different sets of notes, e.g., it can select from outbound links, backlinks, 'folgezettel' chains, siblings (nodes that are linked to from nodes that link to zettel I'm viewing). I probably couldn't use my system at all without these mapped to hotkeys.

Hotkey for backlinks has made tags almost entirely redundant for me. For example, I have an empty zettel called Knowledge Component and every note about that topic is linked there. Then, when I want to see all of them, I navigate to Knowledge Component and browse the backlinks.

What I'm missing is that this does not show context of the links, which might be nice thing to have.

MikeBraddock

Is there maybe a workflow alternative that could facilitate this functionality. It occurred to me that you might assign a secondary UID to the structured. Embed it in the notes listed in the structured note or list of notes.

This UID would only be used as an internal link between the notes and never used to link externally outside the structure you are defining and assigning the additional UID to.

You could search that UID as just a string or use it as a link. It doesn't help with existing structured notes but might be a way to assign UIDs to structures instead of notes.

Or use it as a uid tag or a link

• #202004100903 - Id for the banana nut bread thought network
• [[202004100903]] - Id for the banana nut bread thought network

Might need some quailifier to make these types of UIDs appear different to provide a visual cue that it is an intenal linking UID.

• #IL-202004100903 - Id for the banana nut bread thought network
• [[IL-202004100903]] - Id for the banana nut bread thought network

IL = Internal Link.

The key would be to only use it internally to the structure/network as a convention to keep linking outside of the network out.

Like I said it doesn't help with existing notes that don't already have this.
I haven't tried this but I feel pretty sure it could work.
I will experiment when I have some more time.

Always more than one way to see, think, or do things.
The forum is rich with ideas.

ctietze

Poof, gone is selecta to switch between projects in the shell in favor of fzf

proj() {
  cd "$(find ~/Apps ~/Websites ~/Writing -maxdepth 1 -type d | fzf)"
}

ctietze

@MikeBraddock Like manually placing backlinks, having to do this on my own is error-prone, and the results can be incomplete and/or inconsistent. So I'd favor a (reasonable simple) automatic approach, or not try to use something like it at all

MikeBraddock

@ctietze said:
@MikeBraddock Like manually placing backlinks, having to do this on my own is error-prone, and the results can be incomplete and/or inconsistent. So I'd favor a (reasonable simple) automatic approach, or not try to use something like it at all

Totally makes sense.

ctietze

Here is an underwhelming video demo with explanation:

xal

@JonJia said:
Good idea!

To extend, I think that only need to add a "open in new Tab" to the current "ZID link", it can also solve this problem

Agree. With regard to UX, "open in new tab" seems to be the simplest solution. For me, it is a must have feature. Zettelkasten is like Web. Links are essential. Imagine a Web browser without the possibility to open in a new tab...

ctietze

Yeah, that's a good point. And me, Mr No-Need-For-Special-Features, didn't even think that opening in a new tab could solve the same problem!

Will

@ctietze I working on a Keyboard Maestro macro for this. So far I've got an output like yours that can be pasted into the Omni Bar but the results in The Archive note list are not ideal. A UID is used in all the links to the note as well as in the note itself. It wasn't clear at first but is obvious when you have just one UID from a structure note. I would work to review all the notes all the way through out the zettelkasten but will not help if you want to review depth (1). If we figure this out it might be a good option to add to saved searches. Imagine being able to populate the note list specifying the search depth. Some searches being deeper than others.

See example.

Will

Sorry @ctietze, I initially misunderstood your idea, as evidenced by the post above. As I developed this more I think I got what you are after. This macro will populate the note list a list of all the outgoing and all the incoming notes from and to the target note. This allows in-depth reviews and is easily transversed merely by selecting each note or using the up and down arrow key.

Sorting is a bit of a sticky wicket. I can sort the list of UID's and they get placed in the Omni Bar correctly but the resulting list does not reflect the order the UID's appear. The sort order of The Archive takes precedence. It would be great to be able to sort the list via the input from the macro. Sorting incoming from outgoing or maybe structure note first depth (0), then depth (1), then depth (2) - etc. Maybe there is a reason why this behavior is not allowed which I don't understand.

Another question is when the macro is executed, the editor window goes "Nothing Selected". It would be nice if some note was selected. Preferably the original target note initially.

★★★★★★★★★★★

Here is the macro - download at the bottom of this message.

Here is the Structure Note Review macro.

ctietze

That's great, thanks for sharing!

Just to make sure we're on the same page: I intentionally search for "# 202004110708" OR instead of 202004110708 OR so I only get the notes themselves (since they are supposed to start with a level-1 Markdown heading; folks with YAML headers might use "id: 202004110708" OR and such).

I didn't find the result of just OR'ing Ids to be as useful (202004110710 OR 2019120416145), because they show all notes that link to each of the Ids from the search term, and this could get prohibitively large. Experimentation time

Will

@ctietze said:

Thanks for the challenge. I love working on these little adjuncts. This one caused a little rift between me and my sweetie as she had to call me to help make dinner 3 times because I was deep in it.

Just to make sure we're on the same page: I intentionally search for "# 202004110708" OR instead of 202004110708 OR so I only get the notes themselves

Yes, I see that this would be more useful if it presented a list of the links of depth = 0. The list would be much more focused and manageable for review. Casting a broad net though might reveal unexpected meaning and magic.

(since they are supposed to start with a level-1 Markdown heading; folks with YAML headers might use "id: 202004110708" OR and such).

Picture Will slapping forehead.
Now you tell me my notes are supposed to start with a level-1 Markdown heading that includes the UID. I don't remember why, but early on I jettisoned the UID from the start if a note. I probably had some high fluting ideas about freedom and my perfect sense of aesthetics. So all my notes start with a Level-1 Markdown heading but it is the note title with no UID. Each note does contain the UID, but it appears as a link on a line all by itself with no other characters. My bad. My confusion. This might be something Sasha emphasizes in his class/book.

You probably did tell us that we are supposed to start our not with the header # UID Title but I don't remember it being stress or any reasoning behind it.

Experimentation time

Yes, I see the goal. In order for me to get my zettelkasten in working (for this macro to work) condition, I'll have to apply some bash magic to reapply UID's into the first line of each note. This may take a while.

MikeBraddock

@WIll Oh, the joy of solving problems, positive feedback loops, and flow. Repair that rift. You need someone to remind you to bathe and eat.

ctietze

@Will said:
Yes, I see the goal. In order for me to get my zettelkasten in working (for this macro to work) condition, I'll have to apply some bash magic to reapply UID's into the first line of each note. This may take a while.

I have an idea for a compatible quick fix.

If you're 100% consistent with your footer, you could mass-replace the line with the note's ID and add another "this is the origin, not a link" remark.

It looks like the ID is wrapped in [[...]] like any wiki link, but it's the only thing on its line. In regexp terms: ^\[\[\d{12}\]\]$

The Archive cannot use regexp, so you may need some character at the beginning or end of the line to denote this is not meant to be a link to another note, but the ID of the note itself. Searching through my glyphs for "origin", I found ⌱

[[201905231402]]

Becomes

[[201905231402]]⌱

or some such, and then you can adjust the concatenation of the search string from "# ID" OR to "[[ID]]⌱" OR or whatever else you pick.

Will

@ctietze this has been both challenging and a great regex learning experience. I've become comfortable with Regex Positive and Negative, Look ahead and Look behind, which had confused me in the past. They'll come in handy in the future I'm sure.

I've got this working with this Keyboard Maestro macro. Telling others how to modify the egrep line so that they get the results they want if their notes don't have the level-1 Markdown heading is confusing and hard. A little regex knowledge is all it would take. One more reason to "stay with the class" and not wander off, thinking my way is better. I clearly didn't see this coming. This answers the question "Why are UID necessary/used?" @Sascha nails it.

This should work for you and I've gotten it to work for me by adding a ⌱ in front of the originating link. I included the macro for creating a structure note review list with a depth = 0 and also a macro for the same list only depth = 1.

Your help has been invaluable.

Darryl

@will if your footers are consistent couldn't you just search for the UID, the following square brackets, return character, and the arrow character (sorry, don't know how that would look in regex)?

I'm just wondering because I don't include the UID in the title header in the text file (although it is in the file name itself). I separate the UID so it's clickable (rather than copy and search; I know, I'm lazy):

# Title
ID: [[202004241406]]
Tags: #tag1 #tag2

So searching for all the ID's just becomes, "ID: [[202004241406" OR... All that matters is that I have a consistent, but unique pattern in my headers.

It seems that you have a reproducible, but unique pattern to search for (even though it's a bit longer).

Will

@Darryl said:
@will if your footers are consistent ...

This is the root of my problem, there are at least 8 sofar different footer formats that I've used. I'm embarrassed and am a poster child for why UID's in the body of the note is a good idea and freelance customization can come back to bit you in the ass. This is now solved except for 77 of my earliest notes. None of these notes are structure notes nor are they referred to by other structure notes. On further investigation, I find this not quite to be true. These 77 early notes are an embarrassment. But this is what most of them look like. This is the whole note.

I guess I'll have to think about a bash script or a Keyboard Maestro macro to add the notes link to each of these 77 notes.

But this Structure Note Review macro works for me. I've tweaked it from the one that is available above and can use it for review and it works great.

analogue_man

@Will said:
Here is the Structure Note Review macro.

Hi @Will, as I am a "silent" reader I would like to use this opportunity to thank you for your insights to your workflow and your thinking! I stumbled upon the ZK method (and this forum) earlier this year and since then benefited from your posts. Thank you!

But now to a more relevant issue: As I was surfing the forum I came across your great solution for using Strukturzettel but unfortunately found the link to your macro empty and for some reason, while I was trying to mimic your solution in the screenshots, always receive an error while executing the shell script. Would you mind to reupload the KM macro, please?

Will

Structure Note Review

Here you go. I hope this helps. I don't know what happened to the original.
Let me know if there is any clarification I need to add to the ReadMe.
If I can help with the scripting let me know.

analogue_man

@Will said:
Structure Note Review

Here you go. I hope this helps. I don't know what happened to the original.
Let me know if there is any clarification I need to add to the ReadMe.
If I can help with the scripting let me know.

Excellent! Thank you! It works perfectly! Also: great idea with the › prepending the Zettel id

analogue_man

@Will said:
Structure Note Review

Here you go. I hope this helps. I don't know what happened to the original.

@Will Thank you so much, again! It supports my work with structure notes a lot! I made one addition to the macro as it is only working after I select the structure note in the notes list:

As a first step I added a workaround to copy the file name even when the editor window is focussed (and not the note list). Also, sometimes it might be the case that no note is selected, hence I added a failsafe to avoid accidental note creations via the omnibar:

It's not a perfect solution but I think it's clean enough to work.

Will

@analogue_man, please share your changes and tweaks. I know what you mean by "a workaround to copy the file name even when the editor window is focussed (and not the note list)." the Archive beeps at me when no note is selected, and I've been too lazy (or maybe just more focused on working in my archive) to look at the fix.

I'm not sure I recognize the place of a failsafe. I've never had a Keyboard Maestro create a macro, although I do know the danger of hitting the return key at the wrong time and having an errant note created. I always have blamed myself for this. I have created a workflow where I have never created notes from the Omni Bar, ever. Maybe the first two or three notes I made manually with the Omni Bar, then I made a couple of templates and Keyboard Maestro macros to create notes for my note creation.

analogue_man

Hi > @Will said:

@analogue_man, please share your changes and tweaks.

Certainly! Sorry, I thought the screenshot made it clear.

I needed a solution to activate the note list from within the note, as I like it simple and want to keep everything that distracts me from my focus away from me:

You can rename a note from within the note which moves the focus to the note list. So I activate rename, copy the file name and press Enter (I guess Escape is even more foolproof) to escape the rename action after copying the file name. This is my workaround to please my lazyness.

But there is another problem that probably stems from my personality structure: I like geeky stuff and get inspiration by playing around with it – and your Macro is the perfect candidate! Twice already I found myself running the macro another time with the search results in the Omnibar. So in order to prevent TA from accidentally creating a new note I placed my workaround in an If then-structure that first checks if the conditions for the macro are available.

Everything else stays exactly the same.