Rene's Setevi Converter Script

ctietze · February 2018

This discussion was created from comments split from: Setevi: expandable note link trees.

rene · February 2018

It is coming together. The first prototype gives a #tags and notes overview. Code is on github, requires python3; I didn't create standalone executables yet.

Well, it is a start, but lots of formatting work still needs to be done.

Little example:

[[201802200331]] Note title
The above leads to duplicate note titles when the note id 
is replaced by its title. 

[[201802200331 Note title]]
The above should eliminate that problem if my code 
worked correctly :-)

Due to the nature of Setevi, bulleted lists that contain links probably also deserve a special treatment:

* [[201802191807]] First test note
* [[201802191808]] Second test note
* [[201802191810]] A third note

I have to split text from 'links' to create clickable nodes. That destroys bulleted lists.

So I played with this Zettelkasten:

which can be found on github in the scratch/zksetevi folder; the resulting HTML file is in the output/ directory.

Here are some screenshots:

rene · February 2018

It's much better now that I place links after the paragraph containing them and added a bit of color.

See the difference:

ctietze · February 2018

Cool! Played around with it and am surely looking forward to more of this At the moment, running the conversion script repeatedly changes some connection IDs, but I didn't investigate why this behaves unpredictable at the moment. I guess you'll have that fixes in an hour or two anyway

rene · February 2018

@ctietze Thx. Whaaat? Connection IDs are being changed? Can you elaborate? No need for a github issue but can you provide more details? I will look into the code to check for oddities... Unfortunately and fortunately I have day job, too, so I cannot make any promises regarding how much time I can dedicate to this. So, maybe not in an hour or two, but ...

rene · February 2018

Had time for a little update: citations!

I have pretty long citekeys:

Next up is a tab next to notes and tags, so you can find notes that cite a given source.

But that'll probably have to wait until after dancing class ...

rene · February 2018

@ctietze said:
At the moment, running the conversion script repeatedly changes some connection IDs, but I didn't investigate why this behaves unpredictable at the moment.

Maybe a factor of unpredictability is: When you link to a note id and you have multiple files whose filenames start with that exact note id, then only the first one found will be picked. But I'll dig into this once I can reproduce. The code is probably a bit confusing if you're new to the setevi node structure and how I "did" things, but if you find sth, I'll look at it.

rene · February 2018

A bit of progress:

tags are now also displayed inside notes
citations not only show the source but also other citing notes
there is a "@citations" "tab"/"node" at the top

Here a few screenshots of the current state:

Main nodes

Citations

Notes

Citation open inside note

rene · February 2018

@rene said:

@ctietze said:
At the moment, running the conversion script repeatedly changes some connection IDs, but I didn't investigate why this behaves unpredictable at the moment.

Maybe a factor of unpredictability is: When you link to a note id and you have multiple files whose filenames start with that exact note id, then only the first one found will be picked.

I have to correct myself. This is a python feature: Not all my lists (sets, dicts) are ordered, so the order in which I process nodes is not guaranteed (especially when I iterate over dictionary keys). Hence node_ids can vary from run to run which is reflected in the connections.

I didn't dig into this deeply. If you think the generated HTML links to notes it shouldn't, then please tell me I am wrong .

rene · February 2018

I hope you don't mind the spamming.

When converting my real ZK, I was astonished how cool browsing it with Setevi is (by tag, citekey, related notes...) . The notes look quite OK, and with all the links in place I could really follow my web of thoughts and get carried away. Awesome!

Here is a more realistic example of a ZK note style markdown document:

And even cooler:

ZK on Android

rene · February 2018

On with the spam!

I incorporated pandoc parser support. This gives us tables and fenced code blocks for free.

If you want to try it, make sure you have the pypandoc package installed (which I just discovered, makes life so much easier):

pip install pypandoc  
# or: pip3 install pypandoc, depending on your system

Once I make more progress, I'll generate OS specific executables so nobody has to mess around with python and packages. I want to incorporate more parsers, maybe I'll find a decent multimarkdown capable one.

Also note, that the pandoc parser is rather slow. So don't process hundreds of notes with it if you don't want to wait for a couple of minutes, staring at the progress messages.

Here are 2 sample screen shots:

rene · February 2018

We support local images (via pandoc):

rene · February 2018

Big ones

Fast Multimarkdown parser
Syntax highlighting (pygments) in fenced code blocks

I guess, soon it will be time to create executables so everyone can try it out.

I am also in a discussion with @fbb to switch to a pure vertical layout in Setevi.

Multimarkdown or Pandoc?

The Multimarkdown parser is way faster than spawning a pandoc process.

The script converts pandoc citations (@citekey) and pandoc fenced code blocks (~~~code~~~) to multimarkdown automatically so pandoc can be avoided.

So the only reason I can think of that requires to select the pandoc parser is if you're using pandoc style tables.

To be fair to pandoc: It's still my favourite way to write markdown. It is also still used for creating the sources of citations

Here's what it looks like:

rene · February 2018

I have just published a pre-release!

Now even non-python-freaks can convert their Zettelkästen into the current Setevi format, too! (BTW: Maybe @fbb will update it soon, for a more natural reading experience with a strict vertical layout).

You can download the release from GitHub.

It contains standalone executables for both the macOS and Windows 10 versions of the command line tool. I hope they work on other computers, too.

If anyone's brave enough to try them out, I'd be happy to take feedback whether there are some unresolved dependencies or other problems.

A GUI version will be in the works, soon.

The README has more infos but the quick version is:

3rd party tools

You'll need pandoc installed (creates citation sources).

Usage

usage: zk2setevi.py [-h] [-b FILE] [-e EXTENSION] [-l {single,double,§}]
                    [-p PARSER] [-w MAX_IMG_WIDTH]
                    input_folder output_folder

Convert a Zettelkasten into a Setevi HTML page

positional arguments:
  input_folder          Input: your Zettelkasten folder
  output_folder         Output: Folder to write output HTML to

optional arguments:
  -h, --help            show this help message and exit
  -b FILE, --bibfile FILE
                        .bib file to use for citations if none is in your
                        Zettelkasten folder (default: None)
  -e EXTENSION, --extension EXTENSION
                        extension of your markdown files (default: .md)
  -l {single,double,§}, --linkstyle {single,double,§}
                        link style: double=[[link]], single=[link], §=§link
                        (default: double)
  -p PARSER, --parser PARSER
                        markdown parser: mmd=internal Multimarkdown,
                        pandoc=pandoc, native=native (default: mmd)
  -w MAX_IMG_WIDTH, --max-img-width MAX_IMG_WIDTH
                        maximum width of images (default: 320)

About the available parsers

This tool provides/uses 3 different markdown parsers:

native: the fastest but least capable parser
- handles pandoc and multimarkdown citations
- handles pandoc and multimarkdown style fenced code blocks with syntax coloring
- no tables
- supports images
multimarkdown (default): fast parser, should always be used
- handles pandoc and multimarkdown citations
- handles pandoc and multimarkdown style fenced code blocks with syntax coloring
- handles multimarkdown tables
- supports images
pandoc: slow parser but can handle pandoc markdown
- handles pandoc and multimarkdown citations
- handles pandoc and multimarkdown style fenced code blocks with syntax coloring
- handles multimarkdown tables
- handles pandoc tables
- supports images

You should always use the default mmd parser, even if your text uses pandoc syntax.

The only exception is: If your text contains pandoc style tables, then go for the pandoc parser. Pandoc [@citations] and pandoc ~~~fenced cod blocks~~~ will be converted internally automatically by the tool, so that the multimarkdown parser can handle them.

ctietze · February 2018

Quite some progress, man!

Couldn't the image width be solved with

img { max-width: 100%; width: auto; }

... and surrounding <img> tags in links to open in a new window?

rene · February 2018

@ctietze said:
Quite some progress, man!

Couldn't the image width be solved with
img { max-width: 100%; width: auto; }
... and surrounding <img> tags in links to open in a new window?

Awesome suggestion!!! My HTML is sooooo rusty. You wouldn't believe what a challenge it was for me to figure out how to embed CSS into the body (<div id="scoped-content"><style type="text/css">) for code highlighting

I've just pushed the change!

ctietze · February 2018

Feel free to open an issue in your own repo and ping me if in doubt

rene · February 2018

BOOOOM !!!

We now have a GUI !!!

It will take me a bit to prepare the binary release but this is awesome! Now it's easy to use for anyone

rene · February 2018

Oh, I also renamed the GitHub repository (old references still work but should be updated) to semantic_zk since nobody had claimed it for anything better and I had named the GUI script that way. Goes well with sublime_zk, too

rene · February 2018

Phew !

The binaries for both command line and GUI version are released!

Instructions are in the README

It has nothing to do with this release, but I just like this animation:

Sascha · February 2018

@rene you are beast.

rene · February 2018

MacOS can really be a pain sometimes.

I had to update the binaries of the release for macOS. I didn't want to create a full-blown release for that.

Now the graphical tool is a "real .app" and doesn't open a terminal anymore. Also some PATH issues have been fixed, in order to find pandoc.

In addition, should anything go wrong, a stack trace is reported for debugging.

The new structure of the macOS release is:

macOS:
- semantic_zk-pre-0.1b-macOS.zip containing
  - command_line/zk2setevi (command line tool)
  - semantic_zk-0.1.app (graphical tool)

As before, a few instructions are in the README

rene · February 2018

@Sascha said:
@rene you are beast.

No worries, I can be a beauty, too

rene · February 2018

BOOOM See it in action!

Semantic_ZK is now documented in Zettelkasten style and has converted its own documentation .

See it in action here!

rene · February 2018

New Release!

I've just uploaded a new release.

Now you can specify a base URL if you plan to publish the exported HTML to a remote server. This is necessary to get the image links right:

You can see the converted semantic_zk documentation in action here!

rene · February 2018

Even newer release!

The layout in the generated HTML is now all vertical! Thx @fbb for the setevi update! . This makes for far better readable notes.

You can download the release here

Check out the README for more information.

It's all vertical

ctietze · February 2018

Oh boy, this escalated quickly! Great work, @rene and @fbb!

rene · February 2018

The best release ever!

We now have support to export only part of a Zettelkasten!

We can combine the following filters:

only notes containing specific tags
never notes containing specific tags
only notes not older than date(+time)
only notes not younger than date(+time)

Command Line Example:

zk2setevi docs_zk docs                         \
          --only-tags='#setevi #zettelkasten'  \
          --never-tags='#start'                \
          --from=20180224                      \ 
          --to=20180225

convert the docs_zk folder
write output to the docs folder
only consider notes tagged with either #setevi or #zettelkasten
never consider notes tagged with #start
only consider notes from 2018-02-24 to 2018-02-25 (incl.)

GUI example

Get it here! (Binary Release)

Read it here! (README)

See it here! (Resulting HTML export)

fbb · October 2020

... Setevi is now Sems: http://sems.inetwebspace.de/sems/
(At the moment only in German language)

Zettelkasten Forum