Zettelkasten Forum


Rene's Setevi Converter Script

This discussion was created from comments split from: Setevi: expandable note link trees.

Author at Zettelkasten.de • https://christiantietze.de/

Comments

  • It is coming together. The first prototype gives a #tags and notes overview. Code is on github, requires python3; I didn't create standalone executables yet.

    Well, it is a start, but lots of formatting work still needs to be done.

    Little example:

    [[201802200331]] Note title
    The above leads to duplicate note titles when the note id 
    is replaced by its title. 
    
    [[201802200331 Note title]]
    The above should eliminate that problem if my code 
    worked correctly :-)
    

    Due to the nature of Setevi, bulleted lists that contain links probably also deserve a special treatment:

    * [[201802191807]] First test note
    * [[201802191808]] Second test note
    * [[201802191810]] A third note
    

    I have to split text from 'links' to create clickable nodes. That destroys bulleted lists.

    So I played with this Zettelkasten:

    which can be found on github in the scratch/zksetevi folder; the resulting HTML file is in the output/ directory.

    Here are some screenshots:

  • It's much better now that I place links after the paragraph containing them and added a bit of color.

    See the difference:

    :smile:

  • Cool! Played around with it and am surely looking forward to more of this :) At the moment, running the conversion script repeatedly changes some connection IDs, but I didn't investigate why this behaves unpredictable at the moment. I guess you'll have that fixes in an hour or two anyway :D

    Author at Zettelkasten.de • https://christiantietze.de/

  • @ctietze :smile: Thx. Whaaat? Connection IDs are being changed? Can you elaborate? No need for a github issue but can you provide more details? I will look into the code to check for oddities... Unfortunately and fortunately I have day job, too, so I cannot make any promises regarding how much time I can dedicate to this. So, maybe not in an hour or two, but ... :wink:

  • edited February 2018

    Had time for a little update: citations! :smile:

    I have pretty long citekeys:

    Next up is a tab next to notes and tags, so you can find notes that cite a given source.

    But that'll probably have to wait until after dancing class ... :smiley:

  • @ctietze said:
    At the moment, running the conversion script repeatedly changes some connection IDs, but I didn't investigate why this behaves unpredictable at the moment.

    Maybe a factor of unpredictability is: When you link to a note id and you have multiple files whose filenames start with that exact note id, then only the first one found will be picked. But I'll dig into this once I can reproduce. The code is probably a bit confusing if you're new to the setevi node structure and how I "did" things, but if you find sth, I'll look at it.

  • A bit of progress:

    • tags are now also displayed inside notes
    • citations not only show the source but also other citing notes
    • there is a "@citations" "tab"/"node" at the top

    :smile:

    Here a few screenshots of the current state:

    Main nodes

    Tags

    Citations

    Notes

    Citation open inside note

  • @rene said:

    @ctietze said:
    At the moment, running the conversion script repeatedly changes some connection IDs, but I didn't investigate why this behaves unpredictable at the moment.

    Maybe a factor of unpredictability is: When you link to a note id and you have multiple files whose filenames start with that exact note id, then only the first one found will be picked.

    I have to correct myself. This is a python feature: Not all my lists (sets, dicts) are ordered, so the order in which I process nodes is not guaranteed (especially when I iterate over dictionary keys). Hence node_ids can vary from run to run which is reflected in the connections.

    I didn't dig into this deeply. If you think the generated HTML links to notes it shouldn't, then please tell me I am wrong :smile: .

  • I hope you don't mind the spamming.

    When converting my real ZK, I was astonished how cool browsing it with Setevi is (by tag, citekey, related notes...) . The notes look quite OK, and with all the links in place I could really follow my web of thoughts and get carried away. Awesome!

    Here is a more realistic example of a ZK note style markdown document:

    And even cooler:

    ZK on Android :smile:

  • On with the spam! :smile:

    I incorporated pandoc parser support. This gives us tables and fenced code blocks for free.

    If you want to try it, make sure you have the pypandoc package installed (which I just discovered, makes life so much easier):

    pip install pypandoc  
    # or: pip3 install pypandoc, depending on your system
    

    Once I make more progress, I'll generate OS specific executables so nobody has to mess around with python and packages. I want to incorporate more parsers, maybe I'll find a decent multimarkdown capable one.

    Also note, that the pandoc parser is rather slow. So don't process hundreds of notes with it if you don't want to wait for a couple of minutes, staring at the progress messages.

    Here are 2 sample screen shots:

  • We support local images (via pandoc):

    :smile:

  • Big ones

    • Fast Multimarkdown parser :sunglasses:
    • Syntax highlighting (pygments) in fenced code blocks

    I guess, soon it will be time to create executables so everyone can try it out.

    I am also in a discussion with @fbb to switch to a pure vertical layout in Setevi.

    Multimarkdown or Pandoc?

    The Multimarkdown parser is way faster than spawning a pandoc process.

    The script converts pandoc citations (@citekey) and pandoc fenced code blocks (~~~code~~~) to multimarkdown automatically so pandoc can be avoided.

    So the only reason I can think of that requires to select the pandoc parser is if you're using pandoc style tables.

    To be fair to pandoc: It's still my favourite way to write markdown. It is also still used for creating the sources of citations :smile:

    Here's what it looks like:

    mmd.png 509.4K
  • I have just published a pre-release! :smile:

    Now even non-python-freaks can convert their Zettelkästen into the current Setevi format, too! (BTW: Maybe @fbb will update it soon, for a more natural reading experience with a strict vertical layout).

    You can download the release from GitHub.

    It contains standalone executables for both the macOS and Windows 10 versions of the command line tool. I hope they work on other computers, too.

    If anyone's brave enough to try them out, I'd be happy to take feedback whether there are some unresolved dependencies or other problems.

    A GUI version will be in the works, soon.

    The README has more infos but the quick version is:

    3rd party tools

    You'll need pandoc installed (creates citation sources).

    Usage

    usage: zk2setevi.py [-h] [-b FILE] [-e EXTENSION] [-l {single,double,§}]
                        [-p PARSER] [-w MAX_IMG_WIDTH]
                        input_folder output_folder
    
    Convert a Zettelkasten into a Setevi HTML page
    
    positional arguments:
      input_folder          Input: your Zettelkasten folder
      output_folder         Output: Folder to write output HTML to
    
    optional arguments:
      -h, --help            show this help message and exit
      -b FILE, --bibfile FILE
                            .bib file to use for citations if none is in your
                            Zettelkasten folder (default: None)
      -e EXTENSION, --extension EXTENSION
                            extension of your markdown files (default: .md)
      -l {single,double,§}, --linkstyle {single,double,§}
                            link style: double=[[link]], single=[link], §=§link
                            (default: double)
      -p PARSER, --parser PARSER
                            markdown parser: mmd=internal Multimarkdown,
                            pandoc=pandoc, native=native (default: mmd)
      -w MAX_IMG_WIDTH, --max-img-width MAX_IMG_WIDTH
                            maximum width of images (default: 320)
    

    About the available parsers

    This tool provides/uses 3 different markdown parsers:

    • native: the fastest but least capable parser

      • handles pandoc and multimarkdown citations
      • handles pandoc and multimarkdown style fenced code blocks with syntax coloring
      • no tables
      • supports images
    • multimarkdown (default): fast parser, should always be used

      • handles pandoc and multimarkdown citations
      • handles pandoc and multimarkdown style fenced code blocks with syntax coloring
      • handles multimarkdown tables
      • supports images
    • pandoc: slow parser but can handle pandoc markdown

      • handles pandoc and multimarkdown citations
      • handles pandoc and multimarkdown style fenced code blocks with syntax coloring
      • handles multimarkdown tables
      • handles pandoc tables
      • supports images

    You should always use the default mmd parser, even if your text uses pandoc syntax.

    The only exception is: If your text contains pandoc style tables, then go for the pandoc parser. Pandoc [@citations] and pandoc ~~~fenced cod blocks~~~ will be converted internally automatically by the tool, so that the multimarkdown parser can handle them.

  • Quite some progress, man!

    Couldn't the image width be solved with

    img { max-width: 100%; width: auto; }
    

    ... and surrounding <img> tags in links to open in a new window?

    Author at Zettelkasten.de • https://christiantietze.de/

  • edited February 2018

    @ctietze said:
    Quite some progress, man!

    Couldn't the image width be solved with

    img { max-width: 100%; width: auto; }
    

    ... and surrounding <img> tags in links to open in a new window?

    Awesome suggestion!!! :+1: My HTML is sooooo rusty. You wouldn't believe what a challenge it was for me to figure out how to embed CSS into the body (<div id="scoped-content"><style type="text/css">) for code highlighting :smile:

    I've just pushed the change! :smile:

  • Feel free to open an issue in your own repo and ping me if in doubt :)

    Author at Zettelkasten.de • https://christiantietze.de/

  • BOOOOM !!!

    We now have a GUI !!!

    It will take me a bit to prepare the binary release but this is awesome! Now it's easy to use for anyone :smile:

  • Oh, I also renamed the GitHub repository (old references still work but should be updated) to semantic_zk since nobody had claimed it for anything better :lol: and I had named the GUI script that way. Goes well with sublime_zk, too :smile:

  • Phew :sweat_smile: !

    The binaries for both command line and GUI version are released!

    Instructions are in the README :smile:

    It has nothing to do with this release, but I just like this animation: :smile:

  • @rene you are beast. :worried:

    I am a Zettler

  • edited February 2018

    MacOS can really be a pain sometimes.

    I had to update the binaries of the release for macOS. I didn't want to create a full-blown release for that.

    Now the graphical tool is a "real .app" and doesn't open a terminal anymore. Also some PATH issues have been fixed, in order to find pandoc.

    In addition, should anything go wrong, a stack trace is reported for debugging.

    The new structure of the macOS release is:

    • macOS:
      • semantic_zk-pre-0.1b-macOS.zip containing
        • command_line/zk2setevi (command line tool)
        • semantic_zk-0.1.app (graphical tool)

    As before, a few instructions are in the README

  • @Sascha said:
    @rene you are beast. :worried:

    No worries, I can be a beauty, too :lol:

  • BOOOM :smiley: See it in action!

    Semantic_ZK is now documented in Zettelkasten style and has converted its own documentation :smile:.

    See it in action here!

  • New Release!

    I've just uploaded a new release.

    Now you can specify a base URL if you plan to publish the exported HTML to a remote server. This is necessary to get the image links right:

    You can see the converted semantic_zk documentation in action here! :smile:

  • edited February 2018

    Even newer release! :smile:

    The layout in the generated HTML is now all vertical! Thx @fbb for the setevi update! :+1: . This makes for far better readable notes.

    You can download the release here

    Check out the README for more information.

    It's all vertical :smile:

    xjf.png 411.8K
  • Oh boy, this escalated quickly! Great work, @rene and @fbb!

    Author at Zettelkasten.de • https://christiantietze.de/

  • The best release ever! :smile:

    We now have support to export only part of a Zettelkasten!

    We can combine the following filters:

    • only notes containing specific tags
    • never notes containing specific tags
    • only notes not older than date(+time)
    • only notes not younger than date(+time)

    Command Line Example:

    zk2setevi docs_zk docs                         \
              --only-tags='#setevi #zettelkasten'  \
              --never-tags='#start'                \
              --from=20180224                      \ 
              --to=20180225
    
    • convert the docs_zk folder
    • write output to the docs folder
    • only consider notes tagged with either #setevi or #zettelkasten
    • never consider notes tagged with #start
    • only consider notes from 2018-02-24 to 2018-02-25 (incl.)

    GUI example

    Get it here! (Binary Release)

    Read it here! (README)

    See it here! (Resulting HTML export)

    :wink:

  • fbbfbb
    edited October 2020

    ... Setevi is now Sems: http://sems.inetwebspace.de/sems/
    (At the moment only in German language)

Sign In or Register to comment.