automation software vs scripting support

zk_1000

I am amazed on how much wizardry our forum members can achieve with tools like keyboard maestro (especially @Will). Before finding the forum i didn't know such tools exist. The helper scripts can work system wide and applied on just any software. Compared to this when i write a plug in for an application it only works in this specific tool.

I wonder what are the limitations of this approach? One limitation, i suppose, is that keyboard maestro cannot change how things look like in an app, for example modify themes or syntax highlighting or displays on hover. But as far as text editing goes, anything is possible? What do you hope to achieve with scripting support?

I am still hesitating in learning to use automation tools and prefer to do what i am already familiar with. But in the end i am locked into a Software.

davecan

I'm adding automation to my process because my tool (Obsidian) doesn't do everything I want in the way I want.

It does support plugin development (its an Electron app so its extensible via Node.js) so in theory I could develop a custom plugin to do these things, but that would mean learning Node (ugh) and further tying my workflow to a young proprietary tool.

By building my own external automation support I can remain fairly independent in case Obsidian goes away. I got burned by issues with Evernote, never making that mistake again.

Currently I use external automation as follows:

• Alfred workflow to create a new source note. Given a citekey (from Zotero) it runs a script that fetches the title from the .bib file, queries the Zotero API to get the reference in the correct format (MLA because it seems the easiest to read), creates the source note folder with the name citekey Note title, creates the source note with the same name, pastes in the reference at the top, and pastes in a short template.
• Keyboard Maestro hotkeys that handle marking a new note as a lit note, automatically promoting a literature note to permanent note, etc. On a given note I can press Ctrl-P and it will execute a series of keyboard commands that navigate to the correct place in the title, insert/update the correct label, then run the keyboard commands needed to navigate popup menus and tell Obsidian to move the file to the appropriate folder.

I also have some Alfred workflows that are currently unused but will be again in the near future when I go back to work (I took two months off, yay). For example, one lets me create a meeting note by typing m Name of the meeting from anywhere on my Mac and it creates a note in my work folder named YY.MM.DD Name of the meeting with a brief template included, and executes an Obsidian url scheme that brings Obsidian to the front with the meeting note open. So I can create a new note at the start of a meeting and keep track of action items etc. (this is outside my ZK proper, but in the same Obsidian vault – the ZK is a separate folder containing my source/lit/permanent notes, no pollution of the ZK is allowed)

And I'm also experimenting with creating a command line utility to handle some additional automation tasks instead of having a variety of scripts for them. For example the source note creating script is already migrated into the CLI command, and I'm starting to look into generating statistics from it as well. So I can call zk source <citekey> to create the source note, or zk stats to get generic stats, or zk stats --reading-inbox to get a list of sources in my inbox awaiting processing. Ideally the last one will include some indicator of the degree of completion of processing (unstarted, partial, near-complete, etc) and a priority. This would allow me to generate a prioritized list of items in the reading inbox and begin processing them in priority order, in an effort to move towards true incremental reading as described in this discussion with @gescho a little while ago: https://forum.zettelkasten.de/discussion/comment/11321/#Comment_11321

Ultimately my goal is to bend the app to meet my needs as my frankenkasten continues to grow, and to do so in a way that doesn't strictly tie me too closely to a single note taking app if I can help.

ctietze

@zk_1000 Keyboard Maestro uses the term 'macro' a lot, which is often associated with playing back recorded user input. That level of automation you cannot get with most in-app plug-in systems. (Though if you need to perform an arcane verse of key chords to achieve something and want to record these in a macro, chances are you can bundle the underlying functions in a plug-in as a single command.)

So getting to know some kind of automation tool can be useful for your whole computer life. I don't recommend everyone look into this, because getting to know how these things work requires some tinkering and that's just not everyone's cup of tea. I think the payoffs can be potentially large. -- Scripting the computer reminds me of the days where personal computers booted into a BASIC interpreter, and to run a program from a floppy disk, you had to instruct the computer to execute the load command. This was maybe peak hackability of your work machine. Getting a grip of things that hide below the graphical user interface opens up new possibilities. If you're so inclined, you can begin to learn to think of computerized tasks in different terms. Automating user input, like inserting a time stamp for a key sequence, is just the beginning. Once you open the pandora's box of scripting applications and get comfortable looking up how to do that, you can make them sing together in harmony with a single key stroke

What I personally like about the scripts that are being shared on the forums is their lower-level underpinnings. Since e.g. The Archive doesn't yet support remote-control of the app, our Archons of Scripting have to work on the storage level, on files. So their scripts consist of instructions for the shell that would work just as well when pasted into the Terminal. Learning to think of text and file manipulations as instructions on that level is powerful because these scripts work with any folder of files, independent of the app that creates and maintains the folder.

Enough of the praise of scripting -- It's not for everyone, but scripting on the operating system and file level is super useful on one's way to becoming a power user. App plug-ins on the other hand can help expose context-specific things in a straight-forward way, like "list of all tags" in a note-taking app and make achieving something on this higher level of abstraction easier; with lower-level scripting, you'd have to define the 'tag finding' rules and aggregate info from all the files to model the concept of "list of all tags" yourself.

msteffens

App plug-ins on the other hand can help expose context-specific things in a straight-forward way, like "list of all tags" in a note-taking app and make achieving something on this higher level of abstraction easier

This!

As @ctietze notes, generic macro tools which operate on the operating system & file level can be very useful. And if they solely work on the Markdown files it may be even possible to reuse their macros across different apps, which is nice.

That said, an app that implements scripting and/or a plugin system can go way further: The scripting/plugin API can offer access to commonly used functionality (like the "list of all tags" that Christian mentioned). Even better, the app can expose the app's entire data model in a readily accessible way (this could look like this). For example, you wouldn't need to parse a file to get or modify its H1 heading or keywords, instead you can get/set these elements directly, e.g. like this:

set title of selected note to "# Hello World"

And this is just a very simple example, the possibilities are endless.

It's also important to note that there's no either/or between macro tools and scripting. Instead, the macro tools can easily tap into the scripting API and benefit from it.

davecan

@ctietze said:
Learning to think of text and file manipulations as instructions on that level is powerful because these scripts work with any folder of files, independent of the app that creates and maintains the folder.

Exactly. You described this fundamental principle much better than I did!

Although I'm using Obsidian I'm trying to treat it as just a GUI interface to my text files and offload my custom workflow ideas into my own automation scripts to remove dependency on the GUI. This helps move the "business logic" of the workflow out of the proprietary tool, so the app just becomes the means of pointing and clicking and could be replaced with less effort than if the workflow is wholly dependent on the app itself.

That particular GUI app may or may not exist in a few years, or may move in a direction that no longer supports my workflows if I'm too dependent on the app itself. But files and folders have worked the same way for many decades and there is a vast ecosystem of shell commands for manipulating them.

There's another aspect of the power of scripting as well that is not immediately obvious to those who don't use it: the Unix Philosophy (adopted by most shells regardless of OS) makes it possible to create complex behavior by composing a few simple commands. Since each command is generally small and focused on doing one thing very well, and can accept input from and provide output to other programs that are similarly focused, we can chain together seemingly simple commands to produce complex behavior.

https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html

Here's an example command for finding all files added or modified in the past day by querying the git commit logs: (git allows me to automatically capture "snapshots" of all changes in my files on a set schedule and uploads them to a private repository in the cloud for redundancy, and retains logs of every change ever made to every file)

$ git diff --name-status '@{yesterday}' | egrep -i '^M|^A' | egrep -i '\.md$' | sort

This produces output that is easily read by other commands or scripts. I ended up re-implementing this in python for more power, but this gives you the idea. The new command I have that expands on the above looks simply like this:

$ zk changes
$ zk changes --yesterday
$ zk changes --since="3 days ago"

This means I'm not dependent on the Obsidian devs to implement a feature like this.

Additionally by using git myself I avoid paying the Obsidian devs $4 a month to use their in-app snapshot feature which is undoubtedly based on git behind the scenes.

As Christian says this requires that I encode more "business logic" in my own scripts as opposed to just creating higher abstractions on top of the app, but the benefit is my system is less dependent on the app itself.

And of course anything new I learn while doing this is immediately applicable to any other file on my system as well, not just those in the app, meaning I can bend more apps to my will as well.

I'm interested in what others have to say on this as well.

Will

@ctietze, are we going to get an API like this?

@davecan said:
Although I'm using Obsidian I'm trying to treat it as just a GUI interface to my text files and offload my custom workflow ideas into my own automation scripts to remove dependency on the GUI. This helps move the "business logic" of the workflow out of the proprietary tool, so the app just becomes the means of pointing and clicking and could be replaced with less effort than if the workflow is wholly dependent on the app itself.

Nicely said. By "business logic," I mean the practicalities of the care and feeding of a zettelkasten. I like to focus on the practice, less so on theory, and less so on app specificities. Keeping the "business logic" separate from an app is prudent in these fast-moving times. I like this forum because we can discuss the "business logic" of the ZM aside from app specificities.

There's another aspect of the power of scripting as well that is not immediately obvious to those who don't use it: the Unix Philosophy (adopted by most shells regardless of OS) makes it possible to create complex behavior by composing a few simple commands. Since each command is generally small and focused on doing one thing very well, and can accept input from and provide output to other programs that are similarly focused, we can chain together seemingly simple commands to produce complex behavior.

Reminds me of automaticity and the ZKP. Small commands are linked together to make something novel and useful.

Here's an example command for finding all files added or modified in the past day by querying the git commit logs: (git allows me to automatically capture "snapshots" of all changes in my files on a set schedule and uploads them to a private repository in the cloud for redundancy, and retains logs of every change ever made to every file)

$ git diff --name-status '@{yesterday}' | egrep -i '^M|^A' | egrep -i '\.md$' | sort

This produces output that is easily read by other commands or scripts. I ended up re-implementing this in python for more power, but this gives you the idea. The new command I have that expands on the above looks simply like this:

$ zk changes
$ zk changes --yesterday
$ zk changes --since="3 days ago"

This means I'm not dependent on the Obsidian devs to implement a feature like this.

Additionally by using git myself I avoid paying the Obsidian devs $4 a month to use their in-app snapshot feature which is undoubtedly based on git behind the scenes.

@davecan I have a git account and files on my laptop. Tutor us on how we might set up scheduled backups (I assume that is how it is done?) to git and access the git diff command. Please.

And of course anything new I learn while doing this is immediately applicable to any other file on my system as well, not just those in the app, meaning I can bend more apps to my will as well.

Bravo! Iterative learning. Applying what is learned in one "bucket" of knowledge" to other developing areas of interest/expertise.

ctietze

@davecan Oh yeah, please share your script and your schedule setup!

@Will The screenshot shows an AppleScript API, and that's part of what's eventually coming up, yes.

zk_1000

wow, i can really feel your passion reading these replys! Thank you for taking your time to write such elaborate answers.

I like this forum because we can discuss the "business logic" of the ZM aside from app specificities.

This place is special. I entered a whole new world by finding this site.

It's also important to note that there's no either/or between macro tools and scripting. Instead, the macro tools can easily tap into the scripting API and benefit from it.

Unawareness is the most limiting factor . It feels like i am living under a rock.

Unfortunately Mac OS is not an option. I was looking into tools for Linux and installed a few, but nothing there replaces the possibilities for Mac OS apps. This answered many questions (one of them was why one would install both Alfred and Keyboard Maestro ).

@ctietze said:
Keyboard Maestro uses the term 'macro' a lot, which is often associated with playing back recorded user input. That level of automation you cannot get with most in-app plug-in systems.

I think that autokey looks interesting, but it doesn't seem to offer any macro recording and instead heavily focuses on scripting. Do you think it still transmits the same message - or is this somewhat different?

davecan

@Will The scheduled commits occur from an Obsidian plugin that lets users configure commit frequency (I had it down to 5 minutes for a few weeks, far too granular, now its every hour), commit message, etc. It looks like it has been updated recently to allow two-way use of Obsidian with teams, with options to schedule both pushes and pulls and an option to require a pull before a push to ensure everything is synced from main.

I don't need all that, just the auto-commit, which would be easy enough to implement in a script, and I probably will at some point, but for now "it just works" so I haven't bothered.

@ctietze The script is very nascent right now, as I build it out I may share some but it is nowhere near ready right now.

@zk_1000

This place is special. I entered a whole new world by finding this site.

100% agree

I was looking into tools for Linux and installed a few, but nothing there replaces the possibilities for Mac OS apps.

Funny enough Linux is actually more powerful than MacOS when it comes to scripting automation. They both run on the same core Unix style OS though. Apple made its own tweaks to the OS but a lot of basic scripts can run on both platforms because they can both run bash shells.

zk_1000

@zk_1000 said:
I was looking into tools for Linux and installed a few, but nothing there replaces the possibilities for MacOS apps.

I meant that alternative software is either an imitation or has its own identity, or both. We can only ever decide what is the same and what is different from a biased perspecitve. Thinking about it, it is impossible to provide proper judgement to someone else.

The world of Apple is alien to me which is why it feels so impressive.

@zk_1000 said:
Do you think it still transmits the same message - or is this somewhat different?

What one can achieve with a macro can also be scripted. The workflow is quite different, i suppose, in its granularity and perception of tasks. I would have found it interesting to know if you tend to use more often a record and playback function or directly dive into scripting.

Thinking more about it, it doesn't matter.

davecan

The world of Apple is alien to me which is why it feels so impressive.

MacOS is Unix-compliant so its perhaps not as alien as it seems from the outside.

https://www.howtogeek.com/441599/is-macos-unix-and-what-does-that-mean/

macOS is a UNIX 03-compliant operating system certified by The Open Group. It has been since 2007, starting with MAC OS X 10.5. The only exception was Mac OS X 10.7 Lion, but compliance was regained with OS X 10.8 Mountain Lion.

https://en.wikipedia.org/wiki/MacOS

macOS adopted the Unix kernel and inherited technologies developed between 1985 and 1997 at NeXT, the company that Apple co-founder Steve Jobs created after leaving Apple in 1985. Releases from Mac OS X 10.5 Leopard[11] and thereafter are UNIX 03 certified.[12] Apple's mobile operating system, iOS, has been considered a variant of macOS.

The difference is Apple has world-class interface design and product design as its main differentiation from other manufacturers. Those design elements drive "opinionated" products based on extensive human factors research, as opposed to most *nix products that are typically made to be broadly configurable to meet many needs. Apple just scopes it down and makes a lot of decisions for the user, and those decisions are often correct or close enough to correct for most people's needs.

So MacOS and iOS are largely just flavors of Unix with deep product integration and "lickable" candy interfaces.