Graphviz + Ruby script to visualize Zettel network starting at a particular note

ctietze

I uploaded a public script here, which you can run to get graphviz/dot compatible output of link hierarchies:

https://gist.github.com/DivineDominion/b11982ae1ef555720000b362fe17d7af

Usage: extract_associated_zettel.rb [options] FILE
    -d, --depth [DEPTH]              How many levels to traverse before aborting. Default is 10.
    -o, --output [OUTPUT]            OUTPUT file to store the graphviz dot output instead of stdout
    -v, --verbose                    Print debug info to stdout. Only works with -o/--output to re-route result
    -h, --help                       Prints this help

Feel free to chime in and suggest changes to the code!

---

Example Output

dot output of the David Epstein: Range note

$ ruby extract_associated_zettel.rb ~/Archiv/201909171553\ Range\ -\ David\ Epstein.txt
digraph {
  rankdir=LR;
  overlap=false;
  splines=false;
  node [shape=box];
  0[label="201909171553 Range\n- David Epstein.txt\n"];
  1[label="201909171655 Ericsson\n- The Role of Deliberate\nPractice in the\nAcquisition of Expert\nPerformance.txt\n"];
  2[label="201909171706 Concept\nDeliberate practice.txt\n"];
  3[label="201909231546 Deliberate\npractice could be\nimpossible in wicked\nlearning environments.txt\n"];
  4[label="201909231536 Wicked\nlearning environments.txt\n"];
  5[label="201909231550 Kind\nlearning environments.txt\n"];
  6[label="201909231626 Validity\nof task environments\nas per Kahneman\nand Klein.txt"];
  7[label="201909231624 High-validity\nbut uncertain environments\nproduce games of\npossibilities.txt\n"];
  8[label="201105241557 Colvin\n-- Talent is Overrated.md\n"];
  9[label="201909171737 Cardiac\npatients less likely\nto die when cardiologists\nare absent.txt"];
  10[label="201908111523 Chess\ncomputers free resources\nof humans to produce\na strategy.txt"];
  11[label="201908111520 Automation\nfrees cognitive\ncapacity for important\nthings.txt"];
  12[label="201301092057 Difference\nstrategy and tactics.md\n"];
  13[label="201909231725 Giftedness\ndoes not imply genius.txt\n"];
  0 -> 1;
  1 -> 2;
  2 -> 3;
  3 -> 2;
  3 -> 4;
  4 -> 5;
  5 -> 4;
  5 -> 6;
  6 -> 7;
  7 -> 6;
  3 -> 5;
  0 -> 2;
  0 -> 8;
  0 -> 9;
  0 -> 5;
  0 -> 4;
  0 -> 10;
  10 -> 11;
  0 -> 12;
  0 -> 11;
  0 -> 13;
  0 -> 4;
  0 -> 3;
}

Verbose/debug output of link hierarchies as plain text

$ ruby extract_associated_zettel.rb ~/Archiv/201909171553\ Range\ -\ David\ Epstein.txt --verbose --output out.txt
Searching in `/Users/ctm/Archiv`, starting with:
201909171553 Range - David Epstein.txt
  201909171655 Ericsson - The Role of Deliberate Practice in the Acquisition of Expert Performance.txt
    201909171706 Concept Deliberate practice.txt
      201909231546 Deliberate practice could be impossible in wicked learning environments.txt
        201909171706 Concept Deliberate practice.txt (duplicate contents skipped)
        201909231536 Wicked learning environments.txt
          201909231550 Kind learning environments.txt
            201909231536 Wicked learning environments.txt (duplicate contents skipped)
            201909231626 Validity of task environments as per Kahneman and Klein.txt
              201909231624 High-validity but uncertain environments produce games of possibilities.txt
                201909231626 Validity of task environments as per Kahneman and Klein.txt (duplicate contents skipped)
        201909231550 Kind learning environments.txt (duplicate contents skipped)
  201909171706 Concept Deliberate practice.txt (duplicate contents skipped)
  201105241557 Colvin -- Talent is Overrated.md
  201909171737 Cardiac patients less likely to die when cardiologists are absent.txt
  201909231550 Kind learning environments.txt (duplicate contents skipped)
  201909231536 Wicked learning environments.txt (duplicate contents skipped)
  201908111523 Chess computers free resources of humans to produce a strategy.txt
    201908111520 Automation frees cognitive capacity for important things.txt
  201301092057 Difference strategy and tactics.md
  201908111520 Automation frees cognitive capacity for important things.txt (duplicate contents skipped)
  201909231725 Giftedness does not imply genius.txt
  201909231536 Wicked learning environments.txt (duplicate contents skipped)
  201909231546 Deliberate practice could be impossible in wicked learning environments.txt (duplicate contents skipped)

Please note that the -v/--verbose option prints this to the standard output and requires the dot output to be written to a file instead via -o/--output, so the standard output isn't occupied with the graphviz data anymore.

Example graph with neato

At maximum depth (more than the network link depth actually is)

The output of this:

$ ruby extract_associated_zettel.rb \
    ~/Archiv/201909171553\ Range\ -\ David\ Epstein.txt \
    | neato -Tpng -otest.png

... is this PNG image:

Limited depth to direct connections

And limiting the depth to 1:

$ ruby extract_associated_zettel.rb \
    ~/Archiv/201909171553\ Range\ -\ David\ Epstein.txt \
    -d1 \
    | neato -Tpng -otest.png

argonsnorts

This is so great, thanks!

nickmilo22

So...I haven't jumped into the waters yet, but what's the difficulty level on getting this thing set up? I don't know coding. @Will 's Keyboard Maestro macros are about the limit of my abilities.

argonsnorts

I would say intermediate difficulty level, perhaps? It requires knowing a bit about using command line input in terminal, and there are some dependencies (do you have graphviz installed?).

I'm not super adept at computery things, so it took some amount of googling and fiddling on my end to get it working properly. (For example, I had to figure out that graphviz really doesn't like it when you have quotation marks " in file names. Lesson learned.) But I'm sorry to say that I wouldn't know where to begin to actually walk someone through a complete set up.

ctietze

I just updated the script to work with the Ruby version that comes with macOS Mojave. (I apparently used some newer methods that were not available.)

Assuming you're on a Mac, you will need to

1. Download the script (save it to a .rb file)
2. Install graphviz; for that, you need to
   • open the Terminal
   • install the command-line program manager Homebrew: https://brew.sh/
   • still in the Terminal, install graphviz via Homebrew: brew install graphviz
3. Execute the script.

The script execution command is ruby PATH_TO_THE_SCRIPT PATH_TO_THE_ZETTEL. You can type ruby (note the space), then drag & drop your Ruby script file into Terminal. That should input the path for you. Then type another space and drag & drop the Zettel note into the Terminal. Hit enter. If everything succeeds after a short while, the program prints a couple of lines of graphviz code (see above).

To restore the previous command instructions so you don't have to type it again, press the "up" arrow key.

To turn the graphviz code into an image, you need to pipe the output through to graphviz's neato program that produces pretty nice layout. With the restored command, add | (note the spaces around the pipe operator), and then the target command: neato -Tpng -oIMAGE_FILENAME.png

The result should be of the form ruby PATH_TO_THE_SCRIPT PATH_TO_THE_ZETTEL | neato -Tpng -oIMAGE_FILENAME.png

To open the resulting PNG image, you can open the current directory in Finder: open ., and then open the image from there.

argonsnorts

Alright I lied, here's an attempt to walk someone through setup using Keyboard Maestro. Certainly not perfect, or particularly elegant, but perhaps it'll work.

Graph Zettel in Graphviz.kmmacros

nickmilo22

Thanks @ctietze and @argonsnorts ! I look forward to working through those steps and working with the macro.

nickmilo22

Wow, I got it to work. Before adding the limiter, it spit out this:

Then using the limiter as described in the orginal post, it formed something a bit more in line with my hopes:

Vinho

@nickmilo22: You might also be interested in this discussion and this discussion. The KM macro I linked to on September 2nd in the latter discussion can do exactly what Christian's script does (as far as I can see), but does not use Ruby.

johnny

This is great stuff. And I really appriciate the KM experts in here. I got my macros of Will and they are great. But, I cannot not help thinking -could not these things, the visualizer, the lookup of other notes, templates and the most comon things related to the usage of TA be included in the app itself. Its almost as if you have to buy KM in addition to the archive app at this point.

nickmilo22

@vinho I'm going to revisit those posts and put those KM macros to use, thanks for the suggestion!

@johnny hopefully over time our awesome TA team will judiciously incorporate these features in such a way as to avoid feature bloat.

ctietze

@johnny Scripting integration is next on the to-do list, but fixing some unrelated problems stalls progress at the moment. With the scripting stuff, you will be able to replace a lot of the KM macros with in-app script invocations. (Though not all, e.g. not those that perform mouse movement and clicks for you.)

nickmilo22

@vinho I kind of got your KM macros to work. Not perfectly, and not beyond 1 degree, and not after the first time (can't repeat it for some reason). I know I should provide more details, hopefully I will have the time to do so later.

Vinho

@nickmilo22: I'm sorry to hear it doesn't work smoothly – let me know what the problem is when you have more detail. Thinking more about it, the macro I linked to on September 2nd doesn't do exactly what Christian's script does – it won't show any arrows between children notes.
The other macro would produce something similar to Christian's graph – but only if you selected all the notes belonging to the book in the note list of The Archive before you trigger the macro (which would be easy if they all had a book-tag). Also my graphs are made with the dot-programme of Graphviz, not neato as Christian's. If you want to use neato instead, just replace the dot command in the "Execute Shell Script" action at the end of the macro with neato. You then also need to add the line overlap=false; in the action where the .dot-file is created (e.g. under the line that starts with `label=) – otherwise the boxes in the graph will overlap a lot.

SoyunoForinfo

I finally have the ctietze version working and it's brilliant.

I think I'm stating the obvious here, but having this functionality built into the Archive itself (or perhaps a companion app that one of you coding/scripting geniuses could build?) would be amazing. Especially if the maps had active links to edit the zettels in the Archive, like Vinho's version produces.

Use case: pick a zettel, active the map module or companion app, browse the network of linked zettels, click on a zettel to view or edit and have it open in The Archive, make your change, refresh the map and see any new links you just created updated on the map, rinse and repeat.

Come on.... awesome!