Journaling Zettelkasten stats

Will

I'm embarrassed by my first attempt at Zettelkasten Stats. There were limitations I wasn't taking into account. I have a new Keyboard Maestro macro I set up for monitoring my Zettelkasten Stats. You may find this helpful.

I keep my journal in Evernote, separate from my Zettelkasten. Monitoring personal metrics something I enjoy doing. I like to keep tabs on my progress.

With that in mind, I created a macro that shows me total notes and total word count. Also, it shows the titles and links to the notes I've created or edited in the last 24 hours. Seeing the list of current work is great for reminding me of what I've been working on and for a quick check in the morning. I look at the spelling, grammar, and to see if anything in them is confusing after a night's sleep. Using these links lets me quickly review the notes. (Spaced learning)

This macro fits in any app group within Keyboard Maestro, including Global. You can also have the output go to a "Display Window" instead of pasting via clipboard. See below.

To try this out, given you have Keyboard Maestro installed, open a text editor of your choice and execute the trigger. Be sure to change the directory in three places to the one you use and make adjustments to the regex if you have a different file naming convention. Links will only work in apps with the appropriate functionality. The only ones I've tested are Evernote and The Archive, where they work and Atom where they don't work.

Here is a link to the macro.

In Evernote

In The Archive

In "Display text in a window" - here I removed the links which weren't active.

In Atom - links don't work.

mediapathic

I can't get this one to work, it fails on the regex. Can you tell me what your naming convention is, so I can see if that's the issue? Alternately, can you tell me the correct regex for a convention like

201912221840 Places never in darkness.md 

Would be?

Thanks!

Will

@mediapathic I use the exact note naming convention.

My bad for not explaining and making it easy for differences in setups to be incorporated. If you redownload the macro you'll find a few helpful notes reflecting the changes to look out for including this one.

Good news the regex will handle you file names perfectly. A small change is wanted because our paths-to-zettelkasten are different. Because find saves the path to each note, this has to be handled in the regex. The {33} in the regex removes the path. If the path to your zettelkasten has a different number of characters in it, which is likely, you'll have to change the number to {the appropriate number.}

Look in Keyboard Maestro -- Preferences -- Variables -- noteNames and you will see the paths as they are prepended to your note names.

Here is what my noteNames variable looks like.
/Users/will/Dropbox/zettelkasten/201903030457 Don't get trapped by emptiness.md

Thanks for pointing this out. I appreciate you trying this out. This is an important variable between setups that I failed to mention as something that wants changing before use.

I changed the macro download, adding some explanatory notes.

mediapathic

Ok! Couple of things:

• The regex problem makes sense. I thought we could get around this by using the printf flag to find as shown in this SO answer. Turns out, after a bunch of trying, MacOS, being a BSD-alike, does not have printf flag to find. BUT! it does have awk! So:

find ~/Dropbox/Writing/Zettel/org/org-zettel/*.md \! -name '*★*' -type f -mtime -1 | awk -F/ '{print $NF}' | sort -r 

Produces what we want, just the filename without the path. This means we can now not make the user change that.

That said, I don't speak regex well enough to actually fix that. I just have {0} now but I know there's a better way of doing that

• I thought we could set a variable in KM to a path, and then just propagate throughout the rest of it. However, when I try to do this, it doesn't expand the variable, it literally passes %Variable%zkpath% as a path to the shell, which obviously doesn't work. Must be something weird in the way KM handles variables that I'm missing. Oh well, that would have been nice.

Will

@mediapathic

Thanks for the testing and suggestions.
1. I modified the macro to include the awk statement so the Keyboard Maestro variable only gets the file names without the path. You can redownload the macro and try it out.
2. To fix the regex I removed the first part up to the first capture group entirely. ^.{33} Changing the number of characters from 33 to 0 works too.
3. You got me thinking about setting a variable in Keyboard Maestro for the path and there is a way using the syntax $KMVAR_zkpath. I set the variable at the beginning and use it throughout. This does make it easier as a new user only has to change the path once and doesn't have to go searching for all the instances where the path is called.

I really appreciate the feedback as it has inspired me to make this macro better.

mediapathic

Oh, I didn't realize there was a different way of setting a var to pass to the shell, that makes a lot of sense.

The new version works great! But now I've encountered another issue that is I think probably more difficult to fix. I have set mine to give me the files changed in the previous week (-mtime -7), which works fine. But in doing so I have learned that the regex is really fragile. I have a few files that do not follow the standard naming convention (similar to your starred files, but mine take the form -- 001-WIP.md). The regex chokes on these, of course, and throws an error.

This makes sense, but it's a difficult thing for a user to track down (I went on a long wild goose chase about mtime syntax). I think it would be useful if the macro just skipped the regex parsing if the filename doesn't match, just giving the raw filename. I don't know how difficult this would be to implement in KM, it may be enough of a challenge that it's not worth it. But I think maybe a warning to this effect in the READMEs at least is probably good. (Also, you might put an example in the READMEs about how one can change the mtime amount for different numbers of days, but that's just nice, not needed, I think.)

I'm glad I can provide helpful feedback! This has been a good exercise to get my brain warmed up for doing actual work for the past few days, so it's good for me as well. Thanks for your work on this.

Will

@mediapathic yes regex is fragile or more correctly exacting.

I'm still working on this as it is a good point that most people will have at least a few notes with a different naming convention. and who knows maybe I'll start using some different naming convention. What I have so far is -- by changing the following action to --

You will get all the files but no links. The links are problematic as the character set of the file names is different from the required character set of the links. But I think there could be a solution.

Without links.

With broken links. (This is done with a different regex and I'm still working on this problem.)

ctietze

@Will to fix the links, you have 2 options:

1. Surround the link in <...> so the spaces are part of the link. That works for Markdown in general.
2. URL-encode the string to transform e.g. spaces to %20, see e.g. this Python 1-liner

Will

@ctietze thanks for the tutoring.

1. <thearchive://match/201812250852 Large Data Set> works in Markdown apps.
2. thearchive://match/201903081523%20Interstitial%20Gratitude works in The Archive and possibly other Markdown apps. It works from the command line. This format also works in Evernote and is a solution where users use various naming conventions for notes. This method and the one I was trying (escaping the spaces by adding `\ '. Both these methods are less than aesthetically pleasing to my eye. Yet another vote for plain text and Markdown.

I journal in Evernote and paste links to yesterday's note modifications. As part of my "Monk Morning" I review them and have found this next day review helpful.

@mediapathic How might you envision using this? If pasting the previous week's listing to an app different than The Archive, which one? Maybe just viewing the list in a window works for you.

mediapathic

@Will said:

@mediapathic How might you envision using this? If pasting the previous week's listing to an app different than The Archive, which one? Maybe just viewing the list in a window works for you.

I'm actually mostly using this as a tool to see what's been active in the previous week to make sure nothing falls between the cracks. I'm currently actively figuring out methods for juggling multiple large projects, so this is one of several experimental techniques I'm using at the moment, and I can't guarantee I'll be doing it for any length of time.

I'm currently pasting into The Archive into a note which I just overwrite every time I do it. The links are useful there, so I can easily jump to a note if I want to. But, they are not strictly necessary for the way I'm using this as an overview, because I can easily just do a search for a particular note.

It occurs to me that it might be useful to have a periodic list of these things to compare over time, but I think at the moment for my current needs that is overkill.

Will

@mediapathic pasting the results of this macro into The Archive is elegant.

I reuploaded the macro. The same location should work. Here is a link to the macro in case.

I've reworked the macro incorporating @ctietze's suggestions around the links. It turned out easy. I also changed the regex such that it will accept any filename format. It also turned out to be easy.

@mediapathic said:
It occurs to me that it might be useful to have a periodic list of these things to compare over time, but I think at the moment for my current needs that is overkill.

It would be easy to use a "special" naming format for these and add the character set to the purple action replacing the ★ or use the ★ as your signifier. This would prevent the stats notes from appearing in the list. For example, you could make a file and name it ★201912281651-Stats paste the macro in and just keep it instead of overwriting it. Refer to it or subsequent ones when needed. The macro could be modified to automatically create and name the note file. If you are doing this weekly your Zettelkasten won't get too cluttered or the macro could be set to do a cleanup. But who knows how useful knowing what one was working on (or not working on) several months or years ago.

ctietze

Depending on the stuff you gather, maybe a table would be a good fit?

I have a file called Statistics.txt (no ID to show me that this is not a Zettel but in this case a "meta" note) that contains dozens of lines of monthly stats, generated via a recurring launchd script and appended to the end of the file. It reads:

|   Datum    | total | filter |   Worte  |
| ---------- | ----- | ------ | -------- |
| 2012-01-29 |  1463 |   1215 |          |
| 2012-05-22 |  2076 |   1702 |          |
| 2012-07-24 |  2235 |   1857 |          |
| 2012-08-10 |  2295 |   1909 |          |
| 2012-08-27 |  2323 |   1937 |          |
| 2012-11-02 |  2354 |   1973 |   456729 |
| 2012-12-09 |  2437 |   2055 |   465061 |
| 2013-01-01 |  2471 |   2107 |   464570 |
...
| 2019-10-01 |  5312 |   4971 |   782245 |
| 2019-11-01 |  5348 |   5009 |   788197 |
| 2019-12-01 |  5473 |   5134 |   800388 |

(Total is all text files, "filter" is sans Untitled.*..txt and such)

Except for @Will's "worked on this within the last 24hrs" collection, the raw numbers I saw in this discussion should easily fit inside a table like this. Then you can append and have a running list with a history for the important numbers + a more detailed "last month" record or whatever in a timestamped note.

mediapathic

This is great, @Will.

It's working great for me. And I have an edge case (which is probably more for @ctietze because I think it's about the markdown parser)!

Note these two entries in my stats file: (I have added a bit to the macro to prepend a * to the links for readability, it is irrelevant for the issue at hand.

Note that the first one does not get the text as part of the link, because I (I know, stupidly) have a . as part of the name. The period is an artifact of a way that Drafts saves to my Zettel, and I need to fix that, so this is not actually a problem (and, actually, the failure mode here is great because the link winds up being to the UID, which still works). I just wanted to point it out to @ctietze as an edge case in the parser.

That said, this issue made me think that I want links to just be to the UID and ignore the title, because my titles are unstable but UIDs are not. I'm trying to figure out how to do this, but if @will you want to include that as an option feel free

@Will said: (from "Will's workflow", a now separate discussion (ed. @ctietze ))

My workflow can be improved and I'm open to that. That doesn't mean that what I'm doing now is somehow wrong or bad. When I know better I'll do better.

Nomination to add this to the Core Principles of Zettelkasten

Will

Things are slippery. Find all the exceptions is hard. First, there are files without a UID as in your example below. Second, there was the formating requirements for any note name format which @ctietze provided the solution for. Now we see that . cause the linking to barf.

.> @mediapathic said:
I have a few files that do not follow the standard naming convention (similar to your starred files, but mine take the form -- 001-WIP.md).
>

There are other solutions but I came up with the idea of removing all the .'s from the file names where ever they occur. Not sure what other characters that need this treatment.

Screenshot of the fix.

I've again modified the Keyboard Maestro macro by adding the action Execute Shell Script which removes all the offending .'s. The command is sed 's/\.//g'. Edit: these . should be replaced with %2E as this is the url unicode for a period. This will direct the link positively to the right note. sed 's/\./%2E/g'

So far all my notes start with a UID or a ★ (which is handled by the action executing the find). My output is such that I don't need to worry about the text portion of the filename:

Screenshot of my macro.

@mediapathic, I live outside Moscow Idaho. Only an inch of snow. You?

ctietze

I've extracted Will's workflow into a new discussion. I guess the answers will evolve, so why not gather the off-topic reply in a separate thread!

@mediapathic Regarding the parser, it's the trailing dot for some reason. The CommonMark link destination spec mentions this as any text between < ... >, no exceptions.

Will

@ctietze thanks for moving the discussion.

I read CommonMark link destination. spec and see that the link can include all characters. And so it goes!

Below, at first, we see the link fully formed. Then an added . and we see the link deform. This is very predictable and occurs whenever there is a . anywhere in the link that is surrounded '< ... >'. Mostly a visual thing but does affect rather the note is selected and displayed or just listed in the note list.

mediapathic

@Will said:

@mediapathic, I live outside Moscow Idaho. Only an inch of snow. You?

I'm in the San Francisco Bay Area, so I haven't seen snow in years, sadly.

Completely off topic now, but, I looked at your website and your woodwork is amazing. I have long thought that one possible future for me could be a guy who carves spoons, and you are actually doing that. Mad respect for that, sir. When my living situation becomes a bit more stable, I will probably ask you to sell me a spoon or two