Announcing zkviz: Visualize Link Network Between Zettels

alexchabot

I have the pleasure to announce the release of zkviz. A tool to visualize the network of links between zettles. It produces interactive visualizations that highlight the "centrality" of each note. The GitHub repository has all the installation instructions. You can install zkviz from PyPI with pip install zkviz. The repository also includes a Keyboard Maestro macro that creates the visualization based on the notes currently selected in The Archive.

I hope a few people find it useful, and I look forward to your feedback.

pseudoevagrius

This is so great! Thank you for making this!!

Vinho

Very exciting – I'm really looking forward to trying this!!
Unfortunately I got stuck in the installation process: I installed python3 via Homebrew, then I entered the commands

python3 -m venv ~/envs/zkviz
source ~/envs/zkviz/bin/activate
pip install zkviz

The Terminal then got stuck in the installation process:

Does anyone know what went wrong and what to do?

micahredding

Any suggestions on using with plain text files (.txt)? I tried it using the --pattern argument from the command line, but it doesn't seem to work. It can find .md files in the directory, though.

I also tried it with selected notes, using Keyboard Maestro, but no luck.

Vinho

It was actually still running and finished successfully now – don't know if that's to be expected...

achamess

Awesome! Ran without error on MacOS X 10.14.5 Here is what I got. Pr

So how can we profitably use these graphs? I realize I'm the one who asked for this back in the Grakn thread. But I'd like to hear how people will use this to provide insight.

pseudoevagrius

For what it's worth, I'm having trouble getting it to find any of my plaintext files as well.

Will

I too am having the same problems as @pseudoevagrius -- file extension .txt seems ignored.
Even if I use ~/envs/zkviz/bin/zkviz --pattern '*.txt' Terminal reports ...

I'm sorry, I couldn't find any files.

Works great if I change the file extension to .md.

Also having trouble importing KM macro.
Download zkviz/keyboard-maestro/Generate network visualization of notes selected in The Archive using zkviz.kmmacros and Keyboard Maestro reports when importing ...

This is very intriguing and I'd love to try it out. Lots of work evident @alexchabot, Thanks.

alexchabot

Thank you all for your feedback! It's nice to see that it's getting some use.

I'm traveling for the next few days so I won't be able to answer questions but:

1. I know what's wrong with the .txt files. You're all doing the right thing. I find all the files, but when I match on the filenames to find the zettle ID, I actually hardcoded .md so it break everything. I know how to fix it. I'll take care of that next week.
2. For the Macro not being importable, I'm not sure what's wrong. I'll have to check next week.

Keep the comments and ideas coming.

ctietze

I proposed a fix in a pull request: https://github.com/Zettelkasten-Method/zkviz/pull/3

For those of you at home wanting to follow along, just change line 19 according to my diff to this:

PAT_ZK_ID = re.compile(r'^(?P<id>\d+)\s(.*)\.(?!=\.).*$')

The Keyboard Maestro macro imported just fine on my machine, running KM Version 8.2.4

Will

Keyboard Maestro macro imported successfully. Sorry for the distraction. Beginner mistake. Thought I could just 'Save Link as' but no. Once I used the RAW to save, imported great.

Now just need to figure out how to change line 19. I did a sudo find / -name zkviz.py -print and found nothing. Not sure where this file is hiding.

ctietze

It should be where this is pointing to:

pip list -v | grep zkviz

I'd like to suggest downloading a ZIP from GitHub (Or git clone if that's your schtick), then calling pip install -e . in the resulting dir to use a local "development" copy that live-reloads when you edit the files in there.

magnus

Excellent work! Really appreciate it! Some of us non-archive users have the convention of §201906202320 as link convention. Obviously I could change to PAT_LINK = re.compile(r'\§(\d+)') in my local copy, but would you consider adding both § and [[]] or at least some option to add easily?

ctietze

I guess one could make the PAT_LINK into an optional argument when running the program:

...
-i --identifier Pattern used to find links to other notes. Default: [[(\d+)]]

This would open up support for Wikipedia-style links where the title is used ([[(\w)+]]). One would then have to adjust the recognition for file names, though. I'll leave the implementation of all this as an exercise to someone else

Vinho

@alexchabot: Thanks a lot for all the work you've done! I've played around with it a little bit and now feel able to give some feedback.

A minor bug first: The tool doesn't work if the Saved Searches are hidden. It then can't find the selected notes.

And more important general feedback, including some thoughts on what to use the tool for @achamess:

The main reason I was looking forward to this so much is that I thought it would be a good way to quickly get an overview over the link-relations between a group of zettels. This usually happens when I restructure a part of my Zettelkasten by changing one or more notes that then make changes to other notes necessary – usually hierarchy-related. The size of the group of zettels I'm thinking about will usually be 10-20. Another situation is when I add a new zettel and want to find notes around the same topic that should link to the new zettel. Or: I want to check if there are any zettels in a group that are not integrated properly into the network by links. The "groups" I usually form by using tags or combinations of terms I search for.

With your implementation of the visualisation I unfortunately find that this doesn't work well – mainly because it was made for a much larger number of zettels. In order to deal with a larger number, you had to hide the zettel-titles in the graph, which for my purpose is what makes the tool almost unusable because I'm not able to quickly get an overview over relations anymore. I would have to constantly move the cursor between different zettels to reveal their titles, always keeping the position of previously shown titles in mind. In addition, titles of zettels on a low hierarchy-level are not even shown completely when I move the cursor on them – the appearing title box just shows the zettel-ID as the first part of the title. For my purposes that just doesn't work...

For me, a graph like the one shown in your post from March 7th in the original thread would be appropriate:

All the titles can be seen immediately, which is no problem for a smaller number of notes. An interesting addition could be your different colours for different hierarchy levels, the option to fade out lower hierarchy-levels and maybe frames around the titles for clearer demarcation, but those would just be nice non-essential extras

Thank you again for all your work, I feel bad for not being able to give more enthusiastic feedback

Vinho

Just remembering another thread on this forum: The visualisation that the app contexted offers would be perfect for me somehow integrated into my zettelkasten-editing-app. Unfortunately, that app doesn't work with plain text files and has some other drawbacks compared to The Archive...

alexchabot

I just made the 1.1.0 release of zkviz. You can upgrade by running pip install -U zkviz from the environment where zkviz is already installed.

1. I fixed the issue with the file extensions. It's now also be possible to specify multiple extensions with --pattern '*.txt' --pattern '*.md'.
2. I changed the algorithm used to generate networks large than a 1000 notes. The other algorithm I used scaled really poorly (O(N^2)) so I used a random algorithm, at least for now.

There are a few things I'm looking into:

1. Alternate link patterns, and how to make them optional in a sane way (I guess I could just let people pass in their own regex.)
2. @vinho That is great feedback. I'll investigate if it's possible (and reasonable) to show titles in the visualization (I'm afraid it won't be). Maybe all this time, I should just have shipped the version with Graphviz. I might have to make it an option...
3. @Vinho Indeed, the note list has to be visible because it uses the Accessibility API to list the notes. Ideally, there would be a proper scripting dictionary for that, I know it's on @ctietze's todo list.
4. Building the plotly plot is really slow. I think I know how to speed it up and I'm looking into it now.

Will

Oh, the joy! Great work. So glad to get this going! Had to wait for the .*txt fix.

Boy, this is busy. The KM macro should help by limiting the scope. Unfortunately, I can't figure out the syntax for adding the --pattern '*.txt' argument in the execution script. I get a complaint that seems like it the script understands the selection but then looks for a file with the .md extension.

I assume the variable must be put in the --

    # Build the arguments to 
    args = [zkviz, '--output', output_filename] + filenames

Vinho

@alexchabot: Thanks for your comments – looking forward to what might come
Regarding the "bug" with the hidden sidebar: At the moment not only the note list needs to be visible, also the Saved Searches (they can be hidden separately).

alexchabot

@Will

Boy, this is busy. The KM macro should help by limiting the scope. Unfortunately, I can't figure out the syntax for adding the --pattern '*.txt' argument in the execution script. I get a complaint that seems like it the script understands the selection but then looks for a file with the .md extension.

In my test, you don't need to change the macro. Updating zkviz itself should be enough.

alexchabot

@Vinho

Regarding the "bug" with the hidden sidebar: At the moment not only the note list needs to be visible, also the Saved Searches (they can be hidden separately).

Right. I filed an issue on Github. Again, it's because of the Accessibility API.

alexchabot

📯 I just released zkviz v1.2.0, which includes the option to use Graphviz to render the network. The rendering is much faster and the layout is actually much better (I think). You'll need to install Graphviz separately. I added instructions to the readme on how to do that.

You can upgrade by running pip install -U zkviz from the environment where zkviz is already installed.

There's a new Keyboard Maestro macro, so you'll need to download the latest one from Github and replace your existing one. It includes an option to use Graphviz instead of Plotly.

ctietze

🎉 Great news! Am looking forward to have people post actual images on this forum

Vinho

@alexchabot: Just tried it and find it far better than with Plotly – now it makes sense for me to use it. Many, many thanks! 🎉

Two small things:

1. At the moment the visualisation not only shows the selected zettels, but also the ones that they link to (just the IDs, not the titles). I would quite like to have the option to not show them, that would make the visualisation a bit tidier.
2. My zettels always include their own ID in brackets at the top, so in the visualisation every zettel has an arrow pointing to itself, which makes it more cluttered than necessary. Is there an easy way to not show these arrows?

alexchabot

I just released zkviz v1.3.0, which includes options to hide self-reference links and links to external zettels (as requested by @Vinho).

Upgrade with pip install -U zkviz.

Vinho

@alexchabot: Thanks a lot once again – that is great!

Vinho

One more suggestion (Sorry! ): Option to hide the zettel-IDs in the graph, i.e. to show just the titles. Again this makes the graph a bit tidier and in the visualisation itself I don't find the IDs important.

argonsnorts

Well ok then!

Vinho

@argonsnorts said:
Well ok then!

argonsnorts

Just to follow up, I think this is an awesome visualization tool, @alexchabot. It's helpful to see the cloud clusters that represent certain interlinked ideas. For example, the big grouping on the bottom left is general research for my phd; the smaller cluster above it is research for a single chapter. I can also see a few areas of density beginning to form around ideas I'm just starting to explore. It's like a doppler radar readout, watching a storm build. Maybe I can track and compare growth/density changes over time.

I will say, it would be nice to have a clearer view of what's inside each "cloud"---but I have no idea how this kind of thing comes together, so I'm very happily taking it for what it is now, with awe and gratitude to the wizards.

Any others care to share a visualization?

waterglass22

Newbie trying to install on Mojave. Unable to visualize notes. Error below. Not sure where to proceed from here. Is there further customization in the script that I need in Line11 or Line 168? I used the examples to test and have set the $PATH and correct location of the *.md files under the correct folder. Not sure how to proceed from here. Any help and advice will be much appreciated.

Thanks in advance
Tom

MacBook:~ $xxxx ~/envs/zkviz/bin/zkviz --notes-dir ~/Notes
Traceback (most recent call last):
File "/Users/xxxx/envs/zkviz/bin/zkviz", line 11, in
load_entry_point('zkviz==1.3.0', 'console_scripts', 'zkviz')()
File "/Users/xxxx/envs/zkviz/lib/python3.7/site-packages/zkviz/zkviz.py", line 168, in main
raise FileNotFoundError("I'm sorry, I couldn't find any files.")
FileNotFoundError: I'm sorry, I couldn't find any files.