A request for comments on my concept for a Zettelkasten-compatible tool

fnune

Hello there!

I'm new here. Thanks for the wonderful community. I've been a lurker for a while, and I'm trying to create my own Zettelkasten. I'm a tooling addict so I can't get started until I find the perfect thing. I know this is silly, but I'm having fun.

I've created a concept of what a perfect CLI app would look like for me, in which to implement a Zettelkasten. I've mainly drawn inspiration from jrnl.

https://github.com/fnune/kasten

Comments are desired, as I've barely got started programming it, and most of my time was spent on the concept. I found this post a little bit too late, but the discussion is interesting.

ctietze

Welcome aboard! I'm looking forward to see how your SQLite experiments are going. If you want to stress-test the database at some point, here are 10k Markdown files for you
https://github.com/Zettelkasten-Method/10000-markdown-files

fnune

Hey there, thank you! I was a bit discouraged by the lack of comments

I'll definitely be using that data. Do you have any comments about the API and the design of the tool? Since that's what I'm most worried about. I think it would serve my needs but I don't really have a Zettelkasten yet so...

dld

This is similar to the setup that I use, except that I write markdown/tex files directly and then run a process to extract what I've written to a database (and then to serve a private website from the database). As I look through your setup, it seems that it might be a mighty powerful tool when it's complete.

I'm not one for finishing all tooling before getting started. In fact, I typically think that you hardly know what you actually want before you get started. So if I were you, I would prioritize making a minimal core and begin to use it. The most important thing you're missing (at least, from my quick glance) is a convenient way of finding and linking to existing notes. But of course I'm not you, and you should do whatever you think is best.

Good luck with your system!

fnune

Thank you. My planned feature to link to existing notes would happen through autocompletion in LSP-compatible (Language Server Protocol) editors.

So I'd publish a tiny VSCode plugin and a tiny coc-nvim plugin so that you can type in links as autocompletion results.

I still have to build a proof of concept for that, though, I'm just assuming it will work.

ctietze

I find LSP super interesting! We looked into making an Open Source Markdown LSP component for The Archive, but the details looked kinds weird, so we passed for now Would be cool if you end up with a Zettelkasten daemon that provides LSP callbacks.

fnune

Me too, I'm thinking the basic implementations I need are autocompletion of links (picking by title) and "go to definition" to follow a link.

Additional nice-to-haves would be being able to hover a link to see an excerpt of the note. This is also possible via LSP, I think.

sigod

Welcome to the club.

Everyone seems to be making their own tools for Zettelkasten these days.

I'm pleasantly surprised you're using Rust. You might want to checkout sqlx crate. It's quite an interesting alternative to rusqlite.

Nick

@fnune as someone who doesn't really know much about the programming world, I'd first ask you what is the advantage or benefit of using a CLI app over say a traditional program like Evernote?

Also how have you found traditional methods of note taking to be most limiting? (paper, Evernote, etc)

fnune

@sigod

I used Rust because I'm a beginner and this seems like a good project for me at my level of understanding I will check out sqlx, thanks!

@Nick

To the question of what the advantage is of using a CLI app over a traditioanl desktop application, I'd say the answer could take ages to write down. It all boils down to preference, but here are some of the reasons why I like CLI applications better:

• In the CLI, the main way of passing information from one application to another is text. Operating systems that implement POSIX utilities are full of tools that make the work of handling text a breeze in comparison to the GUI equivalent. Text happens to be the most important aspect of a Zettelkasten, too.
• As a programmer, I am used to using the CLI for many things and I am just faster this way, but you don't need to be a programmer to use it or reap its benefits.
• Most CLI application, because they have text as input and output, give you the chance to own your information/data because it's just there, sitting in your drive. I like this philosophy. Some GUI applications do the same, too. See Obsidian.
• CLI applications are interacted with in a relatively standard way. GUI applications adhere to different conventions of how interfaces are built, and I believe they tend to vary more.

As for the second question: I don't really have a Zettelkasten. I'm new to this idea and I would love to have the dedication to do this, but I still haven't got started getting my notes organized, or writing notes in an organized way. So I was never limited by either paper or Evernote, I just decided to write a program because it's fun

lokithus

Hi, I am also interested in this but I have a few questions about using a sql database instead of plain text as a way to store notes. I am not a programmer, I am only curious of how this is done.

• How would you link one note to other?
  By this, I mean, how it's done? do you have to call to each stored note in the database and then search for text in each one? Is it not resource intensive?

• How would you search your notes?
  With text files you can use grep or ripgrep and make a big quick search of the contents in each notes using regex or simple words. I wonder, how can this be done in a sql database? Do you have to load ALL THE NOTES and then search or there is other efficient way.

• Are ids with date necessary anymore?
  AFAIK, looking at your project, you have a filed for date generations. I guess this means the id for the sql entry doesn't need to be something like YYYYMMDDmmss anymore because you already have a field where you specify the generation date.

Like I said before, I am not a programmer and I am mostly a hobbyist with computers but I my biggest questions is how to handle searching inside notes, in real time and without performance problems. I guess 100 notes will be ok, but 10000, 100000, it will be a little different.

Last time I used a sql database was in 2009 with Windows SQL Server (I don't remember the name), so I guess there should be better tools and libraries than 12 years ago.

Thank you for your project and your time and sorry for my bad english, I am not a native english speaker.

TRumnell

@fnune this is an interesting start, but I feel like I'm looking at the functional design rather than how I would use it. Forgive me if I'm obtuse, but I can't seem to find the instructions for how a user would interact with the tool. I assume if you're building this sort of like jrnl, you imagine the user issuing commands like kasten new "This is my note title" or kasten search Stoicism. I would be interested in seeing what you imagine the experience of using the tool to be, and can tell you from experience that this will change your design dramatically.

As for design, I would highly recommend taking a look at how Bear app on MacOs structures their sqlite database for notes. It may confirm some of your design if you haven't scoped it out already, or change the way you decide to do things.

danymat

I would love an LSP server, maybe it's the best way to find my tags quickly and autocomplete them !

danymat

I created a sample repository for lsp support. The aim is to support tag completions, go to links, hover a link to display it's content.

You can see the repo at:
https://github.com/lsp-zettelkasten/lsp-zettelkasten

If you do want to contribute, feel free to dm me so we can work together.

fnune

I haven't worked on Kasten in a long time, and it's very much incomplete.

You can install it by cloning the repository and running:

cargo install --path .

To be able to do that you need the Rust toolchain: https://rustup.rs/

Then, you can run kasten --help:

kasten 0.1.0
A journaling tool with support for linking entries, which can be used to create a Zettelkasten.

USAGE:
    kasten [FLAGS] [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information
    -v, --view       Prints the database

OPTIONS:
    -e, --edit <edit>    Edits one entry by ID
    -i, --id <id>        Prints a single entry using its ID

Just running kasten will open your $EDITOR and insert a new note. You can follow the format described in the README.

I don't know if I have time to continue working on Kasten, but if people here have interest, pull requests are welcome. Maybe I'll gain some motivation and finish up some basic functionality

danymat

Hello there!

I managed to create a Proof Of Concept for a barebones LSP for Zettelkasten (https://github.com/lsp-zettelkasten/lsp-zettelkasten).

At the moment I implemented only the things that doesn't require parsing from a syntax tree.
I implemented the link autocompletion when typing [[, as this video suggests:

I'm currently discussing with an other guy who is leveraging an open source project called neuron in order to implement his version of lsp. If you want to check out the discussion (https://github.com/mickael-menu/zk/issues/22). Maybe the two projects will collide/merge.

The key differences at the moment, are that his lsp require a sort of database in order to retrieve tags, etc. What I intend to do (but lack some understanding) is to create a syntax tree that can recognize very easily and quickly tags and links in files.

Maybe it can be very cool to implement a client for the archive leveraging the lsp, as I saw in earlier posts.

Feel free to discuss and share ideas,
Stay safe, sincerely yours,
Daniel

ctietze

Applause, @danymat! I really think this is a cool concept

What I intend to do (but lack some understanding) is to create a syntax tree that can recognize very easily and quickly tags and links in files.

I'm using LSP servers using Emacs as client, and I believe at least some server processes produce a cache while they run per project. The server processes keep running for a while then, and are not one-off commands.

(I don't know if the LSP server does the directory monitoring or if you need to feed it from the client.)

With that in mind, you can ditch databases on disk that can get out-of-date and can rely on in-memory caches. Even with large note archives, the space you need in memory is negligible.

A very naive approach you'll find online that's still stupid fast and totally viable is to cache all notes as 'haystacks', all lowercased to speed up search, and do the look-up of a (lowercased) 'needle' via C string functions that are very, very fast. (SQLite full text search is actually slower in my experience.)

With that, you can look up tags on-the-fly. For tag indexes, I'd eventually consider using a tag cache as a projection: a hash map of tag-to-filename (or whatever you use to identify cached notes internally).

To get an AST, libMultiMarkdown or cmark with extensions can help. But unlike code, you will end up with mostly just notes, maybe sub-headings in notes. But there's less interesting stuff in an AST (no functions, classes, functions-inside-classes, namespaces, modules, ...)

Therefore, relying on an AST sounds like overkill to me, at least for an early milestone.

danymat

Hello @ctietze , and thank you for your kind words, it means a lot for me!

If I understand well, you propose to add all notes to RAM in lowercased version, and do an iteration in all files in order to find tags? If so, that means I would have to do regexes in order to find tags. I heard that regexes can be slower than syntax tree, please correct me if I'm wrong.

With that, you can look up tags on-the-fly. For tag indexes, I'd eventually consider using a tag cache as a projection: a hash map of tag-to-filename (or whatever you use to identify cached notes

internally).

I dit not completely understand the purpose of this hash map: creating a binding between a filename and the corresponding tags in the filename?

ctietze

I meant this: https://www.tutorialspoint.com/c_standard_library/c_function_strstr.htm

This perform incredibly well.

To index all tags, a regex is as good as anything. The initial indexing of a note shouldn't take long. The result is a collection of all tags you can cache in your LSP server. Update each note as it changes on the disk.

For live-typing to offer auto-completion or search, regex-matching would become too slow, though. So you will want to have all known tags in memory all the time. Actually finding which note has the tag works super well with strstr linked above. The regex is to index all tags once, the strstr to filter all known notes for the occurrence of the search phrase when a link is clicked. (I think I didn't explain that well in my earlier post where it sounds like the strstr match would be good to index all tags in all notes.)