Python Script "Zettelkasting Dashboard" (needs help)

Will

I'm in the early stages of working on a python application that will provide quantitative stats about my zettelkasting habit. It wants to show progress and a measure of rather I'm easing up or baring down. I also want it to show me what's new so I can review and refactor as needed. I call this my ZK Dashboard.

This says nothing about the quality of my zettelkasting. It says nothing about rather or not I am following best practices. This says nothing about its value to my future self. It does help guide me in tracking my activity, my time sitting in the chair zettelkasting, and this is 73.4% of success. Just showing up regularly is the most critical step.

I am learning python as I go. This application is pretty fast (<2sec.), but I'm sure I'm making beginner mistakes that slow the program down. I'd love for you python gurus to peak at my code and offer some tips after you stop laughing.

Payload
zkdashboard.py

Output

Memento Mori
Days since birth: 23773
Days Until I'm 80: 5447
Days of COVID watch: 593

Super Slogan:
 - if I want something I've never had, I've got to do something I've never done

----------------------------------------

527071 Total word count
9232 Total link count
2369 Total zettel count

13 Zettel Proofing
15 2do's

79 Books Processed
56 Blog Posts
166 Articles Processed
8 Poetry of Zettelkasting videos

----------------------------------------

[3 new zettel yesterday.](thearchive://match/›[[20211028).
13 new zettel in 10 days.
183 new zettel in 100 days.
[2 notes created on 20201028](thearchive://match/›[[20201028)
[5 notes created on 20191028](thearchive://match/›[[20191028)

----------------------------------------

Notes created in the last 10 days

10/19/2021 :: [Death, Nothingness, and Subjectivity ](thearchive://match/Death, Nothingness, and Subjectivity 202110192029)
10/19/2021 :: [Sequential Living Follows Checklists ](thearchive://match/Sequential Living Follows Checklists 202110191753)
10/19/2021 :: [Thinking Of A Memorable Night ](thearchive://match/Thinking Of A Memorable Night 202110192011)
10/19/2021 :: [We Are All Trying Our Best ](thearchive://match/We Are All Trying Our Best 202110191720)
10/22/2021 :: [About Trees ](thearchive://match/About Trees 202110220944)
10/22/2021 :: [Beyond Smart Towards Implementing Ideas ](thearchive://match/Beyond Smart Towards Implementing Ideas 202110220757)
10/22/2021 :: [Inter and Intra Application Linking ](thearchive://match/Inter and Intra Application Linking 202110221102)
10/24/2021 :: [Building Reader Personas ](thearchive://match/Building Reader Personas 202110241005)
10/25/2021 :: [Emergence Of Informative Connections ](thearchive://match/Emergence Of Informative Connections 202110252028)
10/27/2021 :: [T-October 27, 2021 ](thearchive://match/T-October 27, 2021 202110270955)
10/27/2021 :: [B-Pulling Your Painting Together ](thearchive://match/B-Pulling Your Painting Together 202110270800)
10/27/2021 :: [G-Drawing And Sketching ](thearchive://match/G-Drawing And Sketching 202110270759)
10/28/2021 :: [B-Better Than Great ](thearchive://match/B-Better Than Great 202110280509)

ctietze

Can say nothing particular about the use of Python here. The only thing that came to mind to shave off some time is trying to replace nested loops that quickly can have quadratic runtime (i.e. a loop over 10 elements with a loop over 5 elements each inside makes for 10*5=50 iterations in total)

So maybe this:

for uuid in sorted(files):
    for filename in files[uuid]:

Not sure why you need these at all; what's inside files[...] exactly? I don't know defaultdict

The overarching loop over uuids can be trimmed by not going over all sorted files but stop early. In my case, years 2009--2021, everything before 2019 could be ignored.

Will

A heartfelt thanks for taking a moment to look at this.

Thank you @ctietze.

for uuid in sorted(files): iterates and looks at each file for several attributes.
for filename in files[uuid]: this gets the name of the note associated with the uuid. This loop matches the uuid with the filename and is used to show the list of notes, with links created in the last ten days. I think I can pull them apart, but will that help with the speed?

I'm not sure how to iterate through the list/dictionary without a for-loop.

what's inside files[...] exactly?

An unsorted list of all the files and the associated uuid
'20210116': [PosixPath('/Users/will/Dropbox/zettelkasten/Transcorporeality 202101160626.md')], ...

I don't know defaultdict

defaultdict stores a list as a dictionary. It will not throw a KeyError "it provides a default value for the key that does not exists."

# Files is a dictionary mapping of a date to the list of files with that date

files = defaultdict(list)
for child in target_dir.iterdir():
    # Skip directories
    if child.is_dir():
        continue
    match = date_pattern.search(child.name)
    # Skip files that do not match the date pattern
    if match is None:
        continue
    file_date = match.group()
    files[file_date].append(child)> @ctietze said:

The overarching loop over uuids can be trimmed by not going over all sorted files but stopping early. In my case, years 2009--2021, everything before 2019 could be ignored.

In my case, I'd want this to be all-inclusive. But you are right. Limiting the scope might make sense in some cases. I'm not sure if this works when tracking the number of books, blog posts, and articles processed in my zettelkasten.

ctietze

I see! I wasn't familiar with what the outer and inner loop would do on a dictionary.

So the inner loop, for filename in files[uuid], effectively only ever has 1 iteration per UUID. Gotcha. Well, then there's no quadratic runtime

Will

I've move this python script to GitHub. It will be safer there. I've already made several tweaks to it. I'm thinking about how I might ship this so an enduser wouldn't have jump through the pythonie hoops to customize this for their zettelkasten and run this.

ZK Dashboard

ctietze

@Will To not force end users to touch the script, that'll bring you to the world of command line arguments and options. The optparse package looks very capable so you can replace the hard-coded target_dir constant: https://docs.python.org/3/library/optparse.html

Will

@ctietze Thanks for the tip. I've explored optparse and see it is soon to be depreciated, replaced by argparse. I set up a branch with the argparse command line switches and it works wonderfully. Controlling three variables requires a horrific command like.

python zkdashboard.py -z /Users/will/Dropbox/zettelkasten/ -c 2 -s "/Users/will/Dropbox/zettelkasten/Super Slogans 202012281549.md"

And there are potentially a dozen variables to control making command line switches very complicated.

I'm exploring Configureparser as it seems like the option that would allow a user to edit a config file once and be good to go. Do you have any experience with creating and reading .ini files?

ctietze

No, never used that. Shell programs (and scripts) usually do well with sane defaults: use the current directory if -z is not provided, for example

ZettelDistraction

@Will said:
@ctietze Thanks for the tip. I've explored optparse and see it is soon to be depreciated, replaced by argparse. I set up a branch with the argparse command line switches and it works wonderfully. Controlling three variables requires a horrific command like.

python zkdashboard.py -z /Users/will/Dropbox/zettelkasten/ -c 2 -s "/Users/will/Dropbox/zettelkasten/Super Slogans 202012281549.md"

I'm exploring Configureparser as it seems like the option that would allow a user to edit a config file once and be good to go. Do you have any experience with creating and reading .ini files?

I was going to suggest argparse, but you're already using it. Haven't checked configparser...will do now. ... After reviewing, the only question is whether argparse and configparser can be usefully combined...which of course has been addressed Developing Command Line Applications with argparse and configparser.

ZettelDistraction

I could make a few suggestions about the code. You might want to break this into functions. Or I could do it. Some of the import statements can be combined. I haven't done pull requests yet. Looks like a fun project. I could see it growing pretty substantially.

Will

@ZettelDistraction welcome aboard. Dive right in. The water is warm and deep.

I'm a beginner. Python is new to me, though I've been shell scripting since the '80s.
I'm also newish to git. I think I have the basics down, both from the command line and using Sublime Merge connected into Sublime Text, which I use for my IDE.

Yes, I, too, think some of the repetitive code would be served by refactoring into functions. I just haven't yet sat still long enough to envision how to write functions. I've been working on the random ss and looking at some configuration helpers for end-users.

Thanks for the link to The Pleasures of Finding Things Out. I didn't think about using both. I just thought it would be easier to use because configparcer is included by default in python.

ZettelDistraction

I've forgotten how many years I've been programming in python. Around two decades. How about starting with a few smaller items on the master branch, just to practice pull requests--which I haven't tried on github?

Will

Sheepish question? Is there something I need to do to give access? Sure start with combining import statements.

ZettelDistraction

You don't have to give me access. We can work so that it's not a matter of trust.

I'll create a fork of woodenzen/zkdb in flengyel/zkdb. In github jargon, your repository is the upstream repository, and my fork repository (to be created) flengyel/zkdb will be the downstream repository. I'm going follow creating a pull request from a fork. Of course github has to complicate its nomenclature by calling the upstream repository the original repository from which the pull requester created the fork, but I'll attempt to clarify any inconsistent terminology.

BTW you'll notice my github hasn't been that active lately, but this is misleading.

ZettelDistraction

@Will

Just to get used to the idea, I forked your repository (this is shorthand for "I created a fork of your repository" and not a reference to some other activity) and issued a pull request without changing anything. I was planning on doing this tomorrow, but somehow I got caught up in it (I used to mentor research computing fellows in programming and computational cluster administration). Now we can see how GitHub responds. I hope my annotations are readable.

Next, I'll make the change I proposed so that there will be something to compare, and then issue a pull request.

ZettelDistraction

Now you have a pull request from me. It will look like this:

ctietze

I personally love seeing how you two are embracing these tools to collaborate. Pull Requests are a bit awkward at first for sure. (I got Stockholm syndromed into liking them I guess)

The old-school alternative is to send .patch files via email or on the forum. But generating these on the command line and applying them is a bit finnicky. Just wanted to point out that this is 100% still a thing.

There's a (somewhat interactive) tutorial if you want to nerd out on this, based around email collaboration: https://git-send-email.io/

ZettelDistraction

@ctietze said:
I personally love seeing how you two are embracing these tools to collaborate. Pull Requests are a bit awkward at first for sure. (I got Stockholm syndromed into liking them I guess)

You were taken hostage and forced to use git pull requests? In the GUI they are mostly straightforward...

The zkdb is an opportunity to learn git collaboration workflows—among other things. This will help to encourage others to contribute.

Just wanted to point out that this is 100% still a thing.

Linus Torvalds said this in a Google talk where he called his audience "ugly and stupid" for using subversion. The talk is amusing.

There's a (somewhat interactive) tutorial if you want to nerd out on this, based around email collaboration: https://git-send-email.io/

Many thanks for this. I'll take this as a hint not to prolong this thread (too much). I'm a little hampered by working under Windows—I much prefer programming under Linux. I'll install git-send-email. In fact I resuscitated my 2007 iMac to run The Archive and to have a linux-like setup. Had to recall the VNC password I set up.

pryley

Here's an example of how you can get the zettelkasten path from The Archive settings: https://github.com/woodenzen/zkdb/issues/3

ZettelDistraction

@pryley said:
Here's an example of how you can get the zettelkasten path from The Archive settings: https://github.com/woodenzen/zkdb/issues/3

Thanks for this!