Long Time No See/ I haven't abandoned you!

mediapathic

Hi folks who I talked with before, and hi new folks who have joined since my hiatus.

So, I haven't been active on the forums for, wow, almost a year now, because around December of last year I took the plunge on going full on with emacs and org-mode. Really, I can only allow myself one all-encompassing mental system to play around with at a time, so I've been focusing on emacs (It is, as some of you know, a way of life no less then Zettelkasten is).

The good news, though, is that I have been in parallel developing a Zettelkasten methodology in org-mode. It's very rough and custom right now, and involves far more work than is really functional, but I am hoping eventually to turn it into an emacs package, or maybe just a block of lisp code one can use. (I see from my notifications that @GreenBeing has been thinking about the same stuff, maybe a collaboration is in order...). I'm not 'officially' announcing this before I have some actual functionality to share with you ( @rene really raised the bar with his Sublime Text work, and now I feel bad making only vague suggestions) but I just wanted to drop in and say that I miss you all and the discussions here.

Hope you are all well,
-- S

Sascha

Emacs and org-mode... Your departure will be forgiven.

mediapathic

@Sascha said:
Emacs and org-mode... Your departure will be forgiven.

I recently introduced someone on reddit to Zetelkasten, they got super excited, and asked "so how do you implement it?" My response was essentially... "not like you want to I'm almost sure."

Sascha

@mediapathic said:
My response was essentially... "not like you want to I'm almost sure."

Basically, this is what I say when people ask me on my self organisation. Sometimes, I believe Emacs should remain unknown. If you infect your life with it great power comes with great suffering. And then I refer to some (not to be named) basic bitch software as an act of mercy.

But seriously: Looking forward to hear from your ZK implementation in org and Emacs.

mediapathic

@Sascha said:
And then I refer to some (not to be named) basic bitch software as an act of mercy.

"It's kind of like taskpaper".

But seriously: Looking forward to hear from your ZK implementation in org and Emacs.

Actually you might have an opinion on the current problem I'm trying to solve. I've been doing links by replacing the default ID property with a zettel-style timestamp. Some more experienced emacs people say this is a terrible idea, and I should be using CUSTOM_ID (I think because they are trying to preserve uniqueness across systems, which is not really a concern for us). But, this doesn't work because a CUSTOM_ID link includes the file name, which breaks when you refile the heading to another file.

Do you (or any other emacs folk) have thoughts on this?

galen

@mediapathic said:
Actually you might have an opinion on the current problem I'm trying to solve. I've been doing links by replacing the default ID property with a zettel-style timestamp. Some more experienced emacs people say this is a terrible idea, and I should be using CUSTOM_ID (I think because they are trying to preserve uniqueness across systems, which is not really a concern for us). But, this doesn't work because a CUSTOM_ID link includes the file name, which breaks when you refile the heading to another file.

Do you (or any other emacs folk) have thoughts on this?

Unless you want to start hacking around with how org mode handles links, you pretty much have to use ID if you want to stick with the general ZK principle of each zettel being in its own file. CUSTOM_ID really just serves to identify a heading within a particular file. The problem is that org-id.el is not designed to be extensible with user-definable ID formats, and it's not clear what assumptions (if any) are made about the ID conforming to one of the two ID formats that 'org-id-method can be set to. So setting org IDs to zettel IDs will probably work, but might stop working at any point. This is probably why you are getting worried looks from fellow emacs users.

I messed around with trying to use org-mode as a ZK for a while, but after a while I stopped because it felt like I was fighting to reconcile two opposing designs. Org is really at its best with relatively few larger files, and a ZK functions best with many small files. In addition, org's built-in search capabilities (in the form of agenda commands) have well-known scaling issues, and whatever interface I use — agenda commands, or deft, or whatever — nothing I ever found in emacs is as smooth and quick as The Archive for exploring my ZK. So I have serious doubts about emacs's capabilities to serve a 10,000-note ZK effectively without some pretty substantial elisp programming, which I was not willing to invest in.

Org is a wonderful outliner and I regularly use it to arrange and process my reading notes into zettels. But I wasn't able to make org function as a ZK in a reasonable amount of time and effort. If you do manage, I'll be very interested to check your work out!

Sascha

Stick to Galen. I learned emacs just enough to handle org for my self organisation.

mediapathic

@galen said:

Unless you want to start hacking around with how org mode handles links, you pretty much have to use ID if you want to stick with the general ZK principle of each zettel being in its own file. CUSTOM_ID really just serves to identify a heading within a particular file.

I don't understand this. Don't you have to create a heading to apply the ID property to? My current implementation is a bunch of files in deft, each having a single top level heading which has an ID property, and that's what I use to link.

I also sometimes link to items within my agenda files, as my attempt to bring the two areas together. Which I realize is philosophically not in line with traditional zettel, but it is an experiment because that seems to work better with the sort of work I do.

The problem is that org-id.el is not designed to be extensible with user-definable ID formats, and it's not clear what assumptions (if any) are made about the ID conforming to one of the two ID formats that 'org-id-method can be set to. So setting org IDs to zettel IDs will probably work, but might stop working at any point. This is probably why you are getting worried looks from fellow emacs users.

Ah, I see. That makes sense.

I messed around with trying to use org-mode as a ZK for a while, but after a while I stopped because it felt like I was fighting to reconcile two opposing designs. Org is really at its best with relatively few larger files,

This is interesting to me because I'm starting to feel the same, but all of the advice I see for people just getting started is "do whatever you want, it doesn't matter."

and a ZK functions best with many small files.

Is this still true if we consider every headline to be a note? I'm not trying to do this yet, but it occurs to me that a per-headline rather than per-note approach might be more compatible with org-mode. Does this actually break the software-agnostic ideal enough to be a concern, do you think?

In addition, org's built-in search capabilities (in the form of agenda commands) have well-known scaling issues, and whatever interface I use — agenda commands, or deft, or whatever — nothing I ever found in emacs is as smooth and quick as The Archive for exploring my ZK. So I have serious doubts about emacs's capabilities to serve a 10,000-note ZK effectively without some pretty substantial elisp programming, which I was not willing to invest in.

I am definitely hitting some scaling issues. I'm at ~2750 files in my zettel and I've already discovered that adding them all to my agenda is a Huge Mistake. Even deft is getting noticeably laggy.

This also points to another issue I'm having in attempting to combine notes and org; yesterday I was searching for something and, I couldn't find it in deft. I wound up using helm-ag on my agenda files and found it. So without pulling the notes into the agenda, there's the danger of a split which negates my desire to synthesize the two. I need to look at helm-ag and see if I can get it to do a recursive search to incorporate agenda and notes, but even if so that seems a halfassed solution at best.

Org is a wonderful outliner and I regularly use it to arrange and process my reading notes into zettels. But I wasn't able to make org function as a ZK in a reasonable amount of time and effort. If you do manage, I'll be very interested to check your work out!

I realize as I'm typing this that I'm echoing what you've said, and this gives me very little hope . But thanks for giving me a chance to type it all out, and I'm so glad to be able to start a dialogue with some folks about this work.

mediapathic

@Sascha , Meta-question: if this actually turns into a fruitful discussion other people might find useful, is it possible to change the title of this post to something more meaningful and move it to a better category? I can't seem to find a way to do so.

mediapathic

I'm tagging @GreenBeing in this discussion because they mentioned an interest in org-mode as zettel as well, and I'd like to get their thoughts if possible.

Sascha

@mediapathic said:
@Sascha , Meta-question: if this actually turns into a fruitful discussion other people might find useful, is it possible to change the title of this post to something more meaningful and move it to a better category? I can't seem to find a way to do so.

Sure. Make a suggestion and I will follow suit.

alex

I would be very interested in a post about or even a package for org-mode.

helm-ag as most other classic unix search tools has the limitation it is line oriented. But usually I use multiple search terms and I want to find all files that contain them in any order at any position. For this deft is great. But for me it's unbearable slow, too. I use mainly use helm-recoll.

Once I also made an elisp function (for helm and ivy) that showed the output of a bash script. This bash script contained one or multiple runs of grep/ag/rg. When using such a function to jump to headings from my text files from different folders on a SSD I didn't notice any delays with over 50mb of text among 2k files (in contrast to deft). Maybe such an approach could be extended to other problems.

and I still want to explore the improved custom links in org-mode 9 ...

ctietze

Agenda has to load all files it managed en bloc, if I recall correctly, so yeah: loading your whole Zettelkasten into main memory and Agenda's list of open files is likely a bad idea

To use 1 file with 2750 headings, each being a note, could work better with some of the software's limitations. The thousands of metadata items required to keep an index of headings should be similar, though, so maybe it won't help at all. I'm not an expert and could be wrong about the headings being kept in an index.

Years ago, Sascha and I figured out that one doesn't have to partition the Zettelkasten into files as a representation in accordance with the actual Zettel content. One could use a single file with headings. But having a 1:1 correspondence between Zettel and file should be easier to grasp. It's not really haptic, yet, because you cannot touch files, but at least you won't be writing yourself into a corner where you need software to sift through megabytes of text in a single file. -- Then again, when you commit to emacs, why should you ever leave Also, Org mode's handling of headings is a rather haptic experience!

galen

@mediapathic said:

@galen said:

Unless you want to start hacking around with how org mode handles links, you pretty much have to use ID if you want to stick with the general ZK principle of each zettel being in its own file. CUSTOM_ID really just serves to identify a heading within a particular file.

I don't understand this. Don't you have to create a heading to apply the ID property to?

What I mean is that, just as you said above, a CUSTOM_ID link will not search outside of the current org file, whereas an ID link will. So using ID is the only way to have stable links, unless you either use bare zettel IDs for all your file names, or start hacking how org resolves links.

My current implementation is a bunch of files in deft, each having a single top level heading which has an ID property, and that's what I use to link.

Yes, that's the most logical way to structure it, as far as I know.

I also sometimes link to items within my agenda files, as my attempt to bring the two areas together. Which I realize is philosophically not in line with traditional zettel, but it is an experiment because that seems to work better with the sort of work I do.

You could do the same with a file:// link in the archive. Maybe not as smooth because you're switching apps, but it works.

and a ZK functions best with many small files.

Is this still true if we consider every headline to be a note? I'm not trying to do this yet, but it occurs to me that a per-headline rather than per-note approach might be more compatible with org-mode.

It most likely would work better as far as org functionality is concerned. You could get stable links with CUSTOM_ID, org-velocity would work as a zettel search etc etc. But you wouldn't be able to easily order your notes based on last modification, for example. And you are still up against the scaling issue. My average zettel size is 600 bytes. That means at 10,000 notes I have to load up a 6 MB text file. Even if org searching and link resolution stays responsive at that size, there are all sorts of things that make a huge text file a pain to work with. Think revision control, accessing the file on other devices, etc.

Does this actually break the software-agnostic ideal enough to be a concern, do you think?

It does for me. Even though org is plain text, it basically locks you into using Emacs (yes, of course you can edit org files with vim or whatever, but links won't work, agendas won't work, exporting to HTML won't work — nothing will work!). Being locked into Emacs doesn't have as many downsides as being locked into a commercial app, but having your ZK as a huge org files means that the only reasonable way to access it is in org mode. If you are happy living in Emacs, that's fine — but I'm not.

I use three different apps to edit my ZK, depending on the needs of the moment (The Archive, FSNotes, and iA Writer). The dumb-as-dirt structure of the ZK makes this possible.

In addition, org's built-in search capabilities (in the form of agenda commands) have well-known scaling issues, and whatever interface I use — agenda commands, or deft, or whatever — nothing I ever found in emacs is as smooth and quick as The Archive for exploring my ZK. So I have serious doubts about emacs's capabilities to serve a 10,000-note ZK effectively without some pretty substantial elisp programming, which I was not willing to invest in.

I am definitely hitting some scaling issues. I'm at ~2750 files in my zettel and I've already discovered that adding them all to my agenda is a Huge Mistake. Even deft is getting noticeably laggy.

I don't believe that any of these elisp packages use a search index for their searching — so you'll be up against linear performance degradation as the archive grows. (Please correct me if I'm wrong!)

This also points to another issue I'm having in attempting to combine notes and org; yesterday I was searching for something and, I couldn't find it in deft. I wound up using helm-ag on my agenda files and found it. So without pulling the notes into the agenda, there's the danger of a split which negates my desire to synthesize the two. I need to look at helm-ag and see if I can get it to do a recursive search to incorporate agenda and notes, but even if so that seems a halfassed solution at best.

Yes, and this gets at my main problem with Emacs. It never quite works just the way I want it to, and the enticement of perfection keeps me hacking at it for incremental improvements that almost without exception will never repay the time invested in them. It's a constant example of the perfect being the enemy of the good. My blurb on The Archive main page reads "The Archive is relatively fiddle-proof, which is one of the reasons I find working with it productive, even if it doesn’t have all the features I want." In the end one of the main points of a plain-text ZK is that its requirements are so minimal that you shouldn't have to spend much time getting it set up.

That said, one of the requirements of a digital ZK is highly responsive search across a large number of files, typically accomplished with a search index. Indexed searching is not a baked-in feature of emacs, probably because linear searching works well enough for most use cases. But it's not good enough for a ZK. This, combined with the fact that there are many other good options for a ZK, is the org-ZK dealbreaker for me.

EFLS

Hello all,

A fellow emacs and org-mode enthousiast here.
And while the ZK philosophy might be to keep everything platform agnostic, I basically live inside of emacs for all of my writing.

I have been working on an extension of the excellent deft package to create a sort of zettelkasten system in emacs (and briefly talked about it before, here on this forum). Above some of you indicate that deft becomes slow as the number of notes grows into the thousands. I haven't noticed any slowing down, but then I only have a couple of hundred notes.
In any case, with my setup I don't need to interact with deft directly, as it works behind the curtains and not via the (sometimes laggy) user interface.
(As an aside, for anyone experiencing deft slowness, try to disable incremental search.)

It is called zetteldeft.
Feel free to check out what I've written: https://efls.github.io/zetteldeft/
The repo itself is hosted on github: https://github.com/EFLS/zetteldeft

As it is my first attempt at writing elisp, I've made sure to include extensive documentation (mainly for myself) in a single org file that contains both introduction, documentation and the code itself.

Some of the features I am most proud of, is a way to quickly export everything containing a specific search item (with zd-org-search-include).
Recently I've been experimenting with graphviz to create a graphical overview of how notes are interlinked (with zd-org-graph-search), with the following result (a very basic example, but you'll get it): https://github.com/EFLS/zetteldeft/blob/master/docs/img/zd-graph.jpg

I've got a PhD to finish in the coming months so unfortunately won't have much time to devote to this project, but I am definitely open to suggestions for future expansions or basic improvements.

And in due time, when zetteldeft improves and becomes available on MELPA, I'll make sure to announce that in a new thread on these forums.

Cheers
EFLS

mediapathic

@EFLS said:

It is called zetteldeft.
Feel free to check out what I've written: https://efls.github.io/zetteldeft/
The repo itself is hosted on github: https://github.com/EFLS/zetteldeft

I love zetteldeft! It's turning out to be the basis for everything I'm doing here. I'm so glad you spoke up on here, I am thinking about either forking or making a PR on zetteldeft to make a few things work better, so I am glad I can just ask you here.

More later when I'm needed elsewhere.

EFLS

@mediapathic said:

I love zetteldeft! It's turning out to be the basis for everything I'm doing here. I'm so glad you spoke up on here, I am thinking about either forking or making a PR on zetteldeft to make a few things work better, so I am glad I can just ask you here.

Thats awesome to hear, thanks. I wrote this for myself but am happy that other people use it (which was the reason to share it in the first place) and look forward to your feedback and contributions.