Inspect and Adapt - Why You Need Empirical Process Control for Your Zettelkasten

Edmund

Empirical process control is a quality control method that involves continuously collecting data and using it to monitor and improve a process. In the context of a Zettelkasten system, empirical process control could be used to continually evaluate and refine the organization and use of the system in order to maximize its effectiveness and efficiency.

Some potential benefits of using empirical process control for a Zettelkasten system include:

• Improved organization: By continuously collecting and analyzing data on how the Zettelkasten system is being used, it may be possible to identify patterns and trends that can be used to improve the organization of the system. This could involve reorganizing existing notes, creating new connections between notes, or adding new categories or tags to help with navigation and search.

• Increased efficiency: Empirical process control can help identify bottlenecks or inefficiencies in the Zettelkasten system, allowing you to make adjustments and improvements to streamline the process.

• Enhanced knowledge management: By continuously collecting data on the use of the Zettelkasten system, it may be possible to identify areas where the system could be used more effectively to manage and share knowledge within an organization.

• Greater transparency: Empirical process control can provide a clear and objective view of how the Zettelkasten system is being used, which can help to build trust and increase transparency within an organization.

Overall, the use of empirical process control can help to optimize the use of a Zettelkasten system and ensure that it is being used effectively and efficiently to support knowledge management and learning.

My question: What are your preferred tools and workflows to support this process?

—
[1] Ahrens, Sönke. How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking, 2022.
[2] Rohde, Mike. Sketchnote Handbook: The Illustrated Guide to Visual Note Taking. Milwaukee: Peachpit Press, 2012.
[3] Schwaber, Ken. Agile Project Management with Scrum. Microsoft Press, 2004.

Edmund

Here is an example how I'm using these principles. With Dataview we have a great tool for Zettelkasten analytics in Obsidian. Step by step I developed a collection of simple queries for a repeated use to inspect and adapt my Zettelkasten process. Here is my list of:

Tools for Gardening
Process
- Fleeting Notes to be processed
- Literature Notes to be processed
- Imported Notes to be processed
- Daily Notes to be processed

Links
- Permanent Notes without in-links
- Permanent Notes without out-links

Tags
- Permanent Notes without tags

Tasks
- Permanent Note tasks to be finished
- Reference tasks to be finished
- Structure tasks to be finished
- Project Note tasks to be finished

Lost and found
- Permanent Notes without any links
- Permanent Notes never modified
- Orphans to be integrated

Thanks to Dataview you can also use this list with actual data as a Zettelkasten Dashboard:

Edmund

Sometimes charts are better than numbers. But my burndown chart [1] shows me what I've never expected: The work remaining is not only very high, it's also growing. :-(

Any simple rule to adapt my process?

Reference
[1] Schwaber, Ken. Agile Project Management with Scrum. Microsoft Press, 2004, 11-12.

Will

@Edmund said:
Empirical process control is a quality control method that involves continuously collecting data and using it to monitor and improve a process. In the context of a Zettelkasten system, empirical process control could be used to continually evaluate and refine the organization and use of the system in order to maximize its effectiveness and efficiency.

@edmund, you are talking about an intermittent and continuous process. Habit formation around the process is key and what is hard to do. We don't see value on a day-to-day basis. The infrequency of the rewards of following this practice takes a while to surface.

What you describe is only a part of the ZK Dashboard. Additionally, the progress of the number of notes, links, and words represents milestones worth noting and celebrating. Tracking the trend of note creation can be a useful measure to maintain motivation and identify any decrease in motivation. Another section might include a view of recent notes wanting attention. Remember to balance this with one's personal life circumstances. When working on onboarding a high-quality source, it will be desirable to take one's time and slow down.

Will

@Edmund said:
Sometimes charts are better than numbers. But my burndown chart [1] shows me what I've never expected: The work remaining is not only very high, it's also growing. :-(

Any simple rule to adapt my process?

The tip I used to attack my burndown was the one suggested by James Clear. It was simple. Take the total number of notes to be worked on. Figure out what a doable workload is per day. Divide the total number by the workload. Then, project out the completion date and add this to the ZK Dashboard, keeping the target date in front of you.

In your case, 1037 notes to be completed. Maybe three notes a day is the workload you'd be willing to dedicate. That is 346 days to completion on Wednesday, October 16, 2024.

I was able to set up a random threesome from my group of incomplete notes which made for lots of serendipity. This was temporarily added to my ZK Dashboard. I had over 600 to start with and at 3 a day. I started in Oct 2022 and had Zero Orphans in May of this year.

The link to last years discussion on this topic - If you asked me last week, I'd have told you I had a dozen or so orphaned zettel. — Zettelkasten Forum

We'd want to stop creating orphans/incomplete notes along the way or else the snake will just be eating its tail.

Sample of dashboard tracker and the three random notes needing attention.
Nov. 5, 2022

May 10, 2023

GeoEng51

@Edmund @Will It seems, from your numbers, that you are very happy to create a placeholder for a thought, fleeting though it may be, but much less happy to capture the thought in its entirety, or at least in a reasonably mature form. I’ve done the same but try to severely limit the practice, (having no more than 3 or 4 incomplete zettels at a time, and no more than about 10 unlinked zettels). I find it’s important to mature the zettle before the idea and/or its significance fades, and it becomes more difficult if not impossible to complete. I realize this is a personal preference. However, how is having a large number of partially baked zettles any different than practicing the collector’s fallacy?

While process control might be helpful in letting you know when something is getting out of hand, what may be even more important is practising self-discipline - insist to yourself that you progress your zettels to a much more mature and linked state before you create new ones. Otherwise, you are just fooling yourself.

ZettelDistraction

My process goes like this—I agree with @GeoEng51
1. I try to create a citation in Zotero the first time I touch a source. That takes care of so-called "literature notes." By definition, these are citations in a reference manager, Zotero, and I don't think of them any other way.
2. If I create a Zettel, I decide where it fits in and make a YAML header consisting of the title: and the reference-section-title: header variables. This operation is a macro—Zettlr calls its macros snippets—with two fields to fill in: the ID, which contains a pre-populated timestamp at the end, and a title, which comes after the ID. The first portion of the ID is an alphanumeric, Luhmann-like Folgezettel, usually starting with a word up to four characters long.
3. The H1 header is a copy of the title: variable.
4. The reference-section-title: variable has the value References.
5. I rarely have stubs and prefer to know in advance what thought I want to express. The body of the Zettel should have a single focus.
6. I make an exception for short, telegraphic Zettels if these contain an unsolved problem or an open question. In a sense, these are "finished" as open questions or problems. What I try to avoid is an incomplete statement.
7. To call these out, the ID of such a Zettel starts with "OPEN"—an example ID is OPEN.0.23.1105.
8. I have a couple of Python scripts to track format errors. I use Absurdian to generate lists of broken links and orphan Zettels. I haven't invested much in process control because my ZK supports a predictive framework for research, which is a kind of process control.

Will

@Edmund,

It seems to me that your "Inspect and Adapt - Dashboard" is a bit too detailed. With so many categories, it creates another decision point of where to focus work, potentially leading to decision fatigue. I'd suggest simplifying the categories to just one that included "notes to be processed," "tasks," and "Lost and Found." Because every note has an unknown potential, I'd suggest opting for selecting notes to "fix" with some random strategy.

By "fix," I mean incrementally moving the needle of the note's fullness, realizing that the needle will never reach 11.

My experience with this was that it quickly became my favorite part of my day. I was stumbling on old ideas and seeing new patterns. The connections between notes grew and grew. Each day was a wonder. I'm considering looking at this workflow again.

I went from 19% of my ZK needing "fixing" to currently having .01% of the notes in my ZK that are in a state requiring "fixing." These are all grouped in my "proofing oven."

Creating notes is the first step in capturing an idea. Some of us have to work harder than others to balance the excitement of the idea capture with the incremental work of refinement. Refining an idea in your Zettelkasten is a neverending process. However, if you're not careful, you may end up spending too much time trying to perfect a single idea and miss out on discovering new, novel, connecting, and supporting ideas. It's essential to strike a balance between perfecting an idea and exploring new ones.

I am grateful to have engaging, multidimensional life's work and the ability to recognize its value.

This is how I spend my time zettelkasting.
* 50% of the time, capturing new ideas
* 50% of the time, refactoring established notes
* 50% of the time, investigating and interrogating ideas
* 50% of the time, sharing experience with others
* 50% of the time, looking at improving my workflow
* 50% of the time, refactoring structure notes
* 50% of the time, writing from notes

Edmund

@Will said:

• Habit formation around the process is key and what is hard to do. We don't see value on a day-to-day basis. The infrequency of the rewards of following this practice takes a while to surface.
  …

• The progress of the number of notes, links, and words represents milestones worth noting and celebrating. Tracking the trend of note creation can be a useful measure to maintain motivation and identify any decrease in motivation.
  …

• When working on onboarding a high-quality source, it will be desirable to take one's time and slow down.

Thank you for your clear statements. And yes, while playing around with numbers I’m still in the process of habit formation. Maintaining motivation is not so easy and taking time and slowing down is hard work for me.

ZettelDistraction

Regarding habits, I have to force myself to take notes with pen and paper--a mistake.

And this was no mere mistake: it was one of the greatest tragedies in history.
-- Rescue Party. Arthur C. Clarke.

I stand corrected. The ZK YAML header "upgrade" is proceeding apace.

A histogram.

Will

Driving off-topic.
Random Graph

Total word count: 1154353
Mean word count: 312
Total number of files: 3694

Edmund

@Will said:
The tip I used to attack my burndown was the one suggested by James Clear. It was simple. Take the total number of notes to be worked on. Figure out what a doable workload is per day. Divide the total number by the workload. Then, project out the completion date and add this to the ZK Dashboard, keeping the target date in front of you.

The link to last years discussion on this topic - If you asked me last week, I'd have told you I had a dozen or so orphaned zettel. — Zettelkasten Forum

Great idea, great post from last year. Yes, I'm a little bit late. But with your help, I'm now able to adapt my unfinished notes. Using the rule of three is really a motivation to start this work of tidyng up. The idea of having a randomized selection also saves some of my mental energy by reducing the decission load ;-)

My script is now adapted. And I added some gamification to my start page:

(Three) apples a day, keep the doctor away. "to eat" is linked to random tasks to be done. :-)

Edmund

@GeoEng51 said:

While process control might be helpful in letting you know when something is getting out of hand, what may be even more important is practising self-discipline - insist to yourself that you progress your zettels to a much more mature and linked state before you create new ones. Otherwise, you are just fooling yourself.

So true! Self-discipline is the magic key. I'm still trying to improve my self-discipline since many decades. Sometimes I have the feeling it is growing, sometimes I'm disappointed. My best idea for feeling save is "process control". I would be happy if I didn't need it. ;-(

Will

I like the way your dashboard is divided up into focused views.
Start | Activities | Analytics | Domains | Tasks | ACE
I might steal this idea.

What is the section "ACE"?

I, too, have a daily quote on my dashboard. I call them slogans. Sometimes, it is simple, like today's, and sometimes, it is more elaborate.

Edmund

@ZettelDistraction said:
My process goes like this—I agree with @GeoEng51

Thank you for sharing your clear process. I hope my own process also will become more simplified. But it's hard work to change a process. ;-(

I have a couple of Python scripts to track format errors. I use Absurdian to generate lists of broken links and orphan Zettels. I haven't invested much in process control because my ZK supports a predictive framework for research, which is a kind of process control.

Yes, programming is a huge help to clean up a Zettelkasten. You are using Python. In Obsidian JavaScript is available when DataView plugin is installed.

Edmund

@Will said:
I went from 19% of my ZK needing "fixing" to currently having .01% of the notes in my ZK that are in a state requiring "fixing." These are all grouped in my "proofing oven."

Unbelievable: Today I just calculated 91% of my ZK needs "fixing". :-(

This is how I spend my time zettelkasting.
* 50% of the time, capturing new ideas
* 50% of the time, refactoring established notes
* 50% of the time, investigating and interrogating ideas
* 50% of the time, sharing experience with others
* 50% of the time, looking at improving my workflow
* 50% of the time, refactoring structure notes
* 50% of the time, writing from notes

Overall: 350% ?!? - My wife sets my time limit for ZK to 20%. ;-)

Edmund

@ZettelDistraction said:
I stand corrected. The ZK YAML header "upgrade" is proceeding apace.

Here is my simplified overview:

Edmund

@Will said:
I like the way your dashboard is divided up into focused views.
Start | Activities | Analytics | Domains | Tasks | ACE
I might steal this idea .

I would be happy! [1] :-)

[1] Kleon, Austin. Steal like an Artist: 10 Things Nobody Told You about Being Creative. Workman Pub., 2012.

What is the section "ACE"?

I used it for some experiments as a MOC on top of my Zettelkasten. I didn't change my folder system like Nick Milo. For me it's still the 6 folder structure from Sönke Ahrens. But as a top level view it works pretty well.

More about ACE from Nick Milo at: https://www.linkingyourthinking.com/lytcon-2023/nick-milo

GeoEng51

@Edmund There is something fundamentally dissatisfying with your stats of 89 "good Notes" and 963 "notes to inspect and adapt". Are you sure you aren't being way too hard on yourself and seeking for perfection before you accept any note into the first category? 95% of my notes are "good notes", but they are not perfect. As I refer back to one of them, I will on occasion upgrade it, indicating that my thinking has changed or progressed, or I believe there a pithier way to state a concept. The other 5% are either admittedly incomplete or still in need of connections to other zettels.

Edmund

@GeoEng51 said:
There is something fundamentally dissatisfying with your stats of 89 "good Notes" and 963 "notes to inspect and adapt". Are you sure you aren't being way too hard on yourself and seeking for perfection before you accept any note into the first category? 95% of my notes are "good notes", but they are not perfect. As I refer back to one of them, I will on occasion upgrade it, indicating that my thinking has changed or progressed, or I believe there a pithier way to state a concept. The other 5% are either admittedly incomplete or still in need of connections to other zettels.

Yes, I'm aware of being way too hard on myself. Before I started to inspect and adapt my Zettelkasten based on data analytics, I was pretty happy with my ZK and my workflow. And I'm still happy. With my:

7 Criteria for Good Notes
1. Notes in predefined folder "3_Permanent Notes"
2. Timestamp "created" in properties section
3. Timestamp "modified" in properties section
4. Lead paragraph (a short description of the idea) in properties section
5. Link to the source (outgoing link) provided
6. Mandatory tag #type used
7. Mandatory tag #theme used

I established only a formal check of my notes. It says nothing about their content.

The tool automaticly scans my results, keeps me motivated to provide a consistent structure of my notes and the tagging system as well. If 8,5% good notes today is my starting point, may be 15% could be a perfect number for the future.

What I like, is the basic structure of the Zettelkasten system. It runs perfectly well, also I'd feed it with imperfect notes and ideas.