Grakn network visualization

ctietze

Found this via @achamess's Google Group: http://grakn.ai/

At first glance, it looks like a pretty nice graph/visualization language (called "Graql"). But the "engine" can also deal with logical expressions (when X then Y) and infer information.

This is build for big data and genome analysis. But why shouldn't it produce some useful output when you feed it an analysis of your Zettelkasten?

I imagine this should work:

• Traverse all notes;
• collect all links ("forward" links) of the note and produce entity relations in Graql format;
• produce a graph with Grakn.

You could vary the entity types and distinguish regular notes and structure notes to make the latter look different in the graph. You could also add all your tags to the graph, similar to the example below, as another entity (like "city") or as attributes (like "name"):

Maybe one can get fancy and add rules like "when a note is not connected to another note then it gets the "orphan" attribute". I bet you can do even more interesting things to group notes by similarity and proximity of tags, and see where one tag cluster crosses its boundaries and links to another cluster's note, or whatever.

Or maybe not! I don't have a clue, yet, but since I cannot play with this for quite some time, I thought you folks might want to.

mediapathic

I briefly played with making a visualization like this using dot language, but this looks much more extensible than my probably-never-actually-realized idea.

magnus

Not too hard to make a gephi-file as well to play with graphs. If you just make a script that searches all files for IDs and make a new line for each hit you can easily convert to many formats.

Dot example:
201712180005 -> 201801241152

Gephi example:
<edge id="606" source="201712180005" target="201801241152" />

Also @EFLS use in Zetteldeft
https://efls.github.io/zetteldeft/#orgdc68570

mediapathic

@magnus said:

Also @EFLS use in Zetteldeft
https://efls.github.io/zetteldeft/#orgdc68570

Oh right! I always forget that he put that in there.

alexchabot

I read this post over lunch and thought it'd be fun to visualize @ctietze 's ba-zettelkasten-justice-for-hedgehogs notes.

I wrote a Python script to scan the notes folder, parse the zettel IDs, titles, and links, and build the network using the Python graphviz package. The results looks like this:

I've also attached a PDF version, if you'd like to zoom in, as well as the script itself. You'll need to change the extension to .py, and to install Graphviz (the tool) and graphviz (the Python package). It's not super flexible but it has some options:

$ python zk_network_viz.py --help
usage: zk_network_viz.py [-h] [--notes-dir NOTES_DIR] [--output OUTPUT]
                         [--style {record,plaintext}] [--pattern PATTERN]
                         [--layout {circo,dot,fdp,neato,osage,patchwork,sfdp,twopi}]

Visualize the notes network of a Zettelkasten. Each arrow represents a link
from one zettel to another. The script assumes that zettels have filenames of
the form "YYYYMMDDHHMM This is a title" and that links have the form
[[YYYYMMDDHHMM]]

optional arguments:
  -h, --help            show this help message and exit
  --notes-dir NOTES_DIR
                        path to folder containin notes. [default: .]
  --output OUTPUT       name of output file. [default: zettel-network]
  --style {record,plaintext}
                        style of each node. [default: record]
  --pattern PATTERN     pattern to match notes. [default: *.md]
  --layout {circo,dot,fdp,neato,osage,patchwork,sfdp,twopi}
                        layout engine used by graphviz. [default: sfdp]

If there's some interest, I could turn this into a pip-installable package.

achamess

Really cool! I like it!
Now, it would be cool to find a way to label and specify those relations. But the visualization is very interesting. Thanks for doing this!

@alexchabot said:
I read this post over lunch and thought it'd be fun to visualize @ctietze 's ba-zettelkasten-justice-for-hedgehogs notes.

I wrote a Python script to scan the notes folder, parse the zettel IDs, titles, and links, and build the network using the Python graphviz package. The results looks like this:

I've also attached a PDF version, if you'd like to zoom in, as well as the script itself. You'll need to change the extension to .py, and to install Graphviz (the tool) and graphviz (the Python package). It's not super flexible but it has some options:

$ python zk_network_viz.py --help
usage: zk_network_viz.py [-h] [--notes-dir NOTES_DIR] [--output OUTPUT]
                         [--style {record,plaintext}] [--pattern PATTERN]
                         [--layout {circo,dot,fdp,neato,osage,patchwork,sfdp,twopi}]

Visualize the notes network of a Zettelkasten. Each arrow represents a link
from one zettel to another. The script assumes that zettels have filenames of
the form "YYYYMMDDHHMM This is a title" and that links have the form
[[YYYYMMDDHHMM]]

optional arguments:
  -h, --help            show this help message and exit
  --notes-dir NOTES_DIR
                        path to folder containin notes. [default: .]
  --output OUTPUT       name of output file. [default: zettel-network]
  --style {record,plaintext}
                        style of each node. [default: record]
  --pattern PATTERN     pattern to match notes. [default: *.md]
  --layout {circo,dot,fdp,neato,osage,patchwork,sfdp,twopi}
                        layout engine used by graphviz. [default: sfdp]

If there's some interest, I could turn this into a pip-installable package.

alexchabot

@achamess What would you label the links with? With markdown links, I could use the link’s text, but there’s no other info in the wiki links. Maybe something taken from the source file?

Vinho

This looks very interesting, thanks @alexchabot for trying and posting this! I would love to have the opportunity to create such a graph from a selected set of notes. Ideally this would be possible with the selection of notes that shows in the sidebar of The Archive after I've searched for something – that makes it easier to get an overview over existing notes on something and to find existing and missing connections between a set of notes.
What would I need to do in order to make that happen? How do I use the script you posted? Could it for example be a macro in Keyboard Maestro, activated by a keyboard shortcut?
As far as I understand, I need to install Homebrew or MacPorts to install Graphviz. I guess I also need to download Python. Is there an easier way to get this visualisation without your script and having to download all these things (that I don't have any other use for at the moment)? Any (free) app that does this maybe?

alexchabot

@Vinho Those are great ideas. It'd definitely be possible to trigger the script from KeyboardMaestro. I'd need to look into getting the list of selected files from The Archive to create the graph, but I assume it's possible. I created an issue in the repo where I'll be sharing the code.

You don't need to install Python, because it comes standard on the Mac, but I used a third-party packages to create the graph because that's what what easiest. As for Graphviz itself, it does look like it's must be installed with Homebrew of Macports (I recommend Homebrew) and there isn't much I can do about that. I don't know of a pure-Python library that can easily make such graphs.

ctietze

@alexchabot Getting the list of selected files could work by scripting against the Accessiblity identifiers of the UI components -- it will be easier once I add a proper AppleScript dictionary later this year.

Vinho

@alexchabot: Would be awesome if that could work. With all these Keyboard Maestro macros The Archive is an incredibly powerful tool...

Vinho

What I would be almost equally happy with and might be easier to implement is the possibility to create such graphs for tags (enter a tag and a network-graph is created for all the notes with that tag).

pseudoevagrius

I would vote for a pip install! Not to give you more work but just to show my interest and support for this great execution of the idea!

alexchabot

It’ll definitely be pip-installable. But i need to figure out if there’s a sane way to make their network graphs without graphviz. Otherwise, graphviz will need to be installed separately (which is a pain). I’m looking into using NetworkX and matplotlib, which should be ok for small graphs.

Vinho

@alexchabot: Any news? If I could help with this in any way I would, but unfortunately my few programming skills won’t be of much use...

alexchabot

@Vinho I've not abandoned! But you know, life got in the way. I just finished working on a version that uses plot.ly instead of Graphviz, which means it'll be much easier to install and we'll get interactivity for free! I'll try to publish it this weekend, but it might have to wait to the weekend after that.

Vinho

@alexchabot Great, very much looking forward to it!

alexchabot

zkviz is live! See the announcement post for more details.

Vinho

@alexchabot:
I'm just trying to do what you did above to ctietze's zettels with a subset of mine using the script you posted, but can't get it to work so far. I'm quite new to all of this, so have a few basic questions:

1. If I download your script and change the extension to .py – where do I have to store the file? Just put it in the folder /usr/local/bin?
2. How do I install Graphviz (the tool)? I executed the command brew install graphviz in the Terminal, but I guess that just installed the Python package?
3. When the tool and the package have been installed, can I then just run the command zk_network_viz.py in the Terminal (with the options in square brackets)? Or do I have to do something else as well?

alexchabot

@vinho I just made a new release of zkviz that supports Graphviz. I recommend using that instead of this script, since zkviz includes a few more options, is better tested, and is more flexible.

Vinho

@alexchabot: Great, thanks a lot for that!

Still would be interested in knowing how I would have gotten the above script to work, though – just to improve my skills

alexchabot

@Vinho said:
1. If I download your script and change the extension to .py – where do I have to store the file? Just put it in the folder /usr/local/bin?

When you type the name of a program in Terminal.app, your shell (probably Bash) looks through a list of folders for that program. You can see the list of folders by executing the echo $PATH command, like so:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

The text below the $ is value of the PATH variable. The folder names are separated by the : character.

If your program, like zk_network_viz.py, is in one of those folders, your shell will be able to find it and you'll be able to just type its name:

$ zk_network_viz.py

If it's not in one of those folders, you'll have to type the full path to the program, like ~/bin/zk_network_viz.py, where ~ means your "home" folder, where Documents, Downloads, Pictures, etc., are, or maybe ~/Dropbox/programs/zk_network_viz.py if you've put it on Dropbox.

You can add folders to the PATH variable by adding a line to your shell's "config file," which is ~/.bash_profile. You can open the file from the Terminal with:

$ open ~/.bash_profile

Add the line below in ~/.bash_profile to prepend the ~/bin folder to the PATH variable.

export PATH="~/bin:$PATH"

And then you'll need to open a new Terminal tab, or restart Terminal for the changes to take effect. After that, if zk_network_viz.py is in your ~/bin folder, you'll be able to execute from any folder.

There's one more thing: wherever you put zk_network_viz.py, you'll have to "mark it as executable" for your shell to treat the script as a program. Otherwise, it's treated as a regular file. You do that by executing the chmod program with the +x option, followed by the path to the file you want to make executable:

$ chmod +x ~/bin/zk_network_viz.py

The +x options is the one that means "make executable". You can remove the executable mode using -x instead of +x.

2. How do I install Graphviz (the tool)? I executed the command brew install graphviz in the Terminal, but I guess that just installed the Python package?

You did the right thing, brew install graphviz installs the graphviz program (actually brew is a "general purpose" package manager. It can install programs written in different languages, and also install actual languages, like Python, or Ruby. I think that Graphviz is itself written in C++.

Each of those languages has, in turn, its own package manager to install libraries (also called packages) for that language. The Python one is called pip, the Ruby one is called gem. When you do pip install graphviz, it installs a Python "wrapper" called graphviz to interact with the program called graphviz (super confusing, I know). Homebrew usually installs programs in the /usr/local/bin folder, which is probably already on your path (the PATH variable).

3. When the tool and the package have been installed, can I then just run the command zk_network_viz.py in the Terminal (with the options in square brackets)? Or do I have to do something else as well?

If zk_network_viz.py is on your PATH, then yes, you'll be able to just type zk_network_viz.py. In the help above the square brackets denote options; things you can specify but don't have to. If you're already in your Notes folder, you can just type:

$ zk_network_viz.py

But if you want to specify a different folder you should use:

$ zk_network_viz.py --notes-dir ~/Dropbox/Notes

I hope that helps!

Vinho

@alexchabot: Wow, huge thanks for the time you put into explaining this – that was much more than expected... I think I followed all your instructions, but unfortunately it still doesn't work. When I try to run $ zk_network_viz.py the file doesn't seem to be read properly. Commands like "import" are not found and some of the comments are interpreted as commands...

Could it have something to do with my version of python being more recent than your's? I had to run pip3 install graphviz instead of pip install graphviz as well because I have Python 3 installed...

alexchabot

@Vinho

It's not your fault, it's because the zk_network_viz.py file doesn't inform your shell (Bash) what program to execute it with. You need to add this line at the top of the file, it's called a shebang line. It needs to be the path to the Python executable where you've installed the requirements (when you did pip3 install graphviz). If you installed graphviz in the default Python that you installed with brew then they shebang line will be:

#!/usr/local/bin/python

If you want to be more general, you can use:

#!/usr/bin/env python

which, according to Wikipedia, will "execute [your program] with a Python interpreter, using the program [you shell] search path to find it".

Vinho

Unfortunately it still doesn't work. Now the error is

line 15, in <module>
    from graphviz import Digraph
ImportError: No module named graphviz

I think I'll just give up the idea of trying to make my own visualisation for now

alexchabot

@Vinho Let's try again.

Use the which command to figure out which Python and which pip is on your PATH. Here's my default output for all 4 commands.

$ which python
/usr/local/bin/python

$ which pip
/usr/local/bin/pip

$ which python3
/usr/local/bin/python3

$ which pip3
/usr/local/bin/pip3

If you installed graphviz with pip install graphviz, then the shebang in zk_network_viz.py should be the output of which python. If you used pip3 install graphviz, then it should be the output of which python3.

Vinho

@alexchabot: It worked with /usr/local/bin/python3, great! Thanks so much for your efforts, that is very kind of you!!

ctietze

@Vinho Can you try with #!/usr/bin/env python3? That should work for people who have their Python 3 installed e.g. in their user/home folder.

Vinho

@ctietze said:
@Vinho Can you try with #!/usr/bin/env python3? That should work for people who have their Python 3 installed e.g. in their user/home folder.

@ctietze: That seems to work as well.