The Archive: Plugin System dev log

ctietze

In the spirit of transparency and buzz and excitement, I want to share with you some progress I'm making on The Archive's plugin system.

---

Starting with an odd one to not spoil all too much at once

Today was the first time I used the very much work-in-progress development view to verify a couple of JavaScript quirks in the interactive REPL. It's basically line-by-line code execution to test algorithms and poke around things:

The top part is the actual script editing part and filled with a placeholder comment. Nothing to see there yet.

So here's a verified rumor -- the scripting language will be JavaScript by default. Reason for that is the excellent and secure environment Apple provides out of the box. (Other languages I would have preferred would require us to ship the language environment or depend on the user installing everything. So this is currently the only sane way to get started. Let's see what the future holds, though.)

Will

Yahoo!

I get to learn another programming language. JavaScript isn't the most popular programming language for no reason. Supposedly easy to understand and comes preinstalled are great attributes. This will make the sharing of plugins a no-brainer. What language do Obsidian, Roam, and Zettlr use?

We'll see how easy JavaScript is to learn. The challenge is locked and loaded.

ctietze

@Will Looking forward to see what you'll be exploring! Coming from Python, the syntax is different, but the core principles and paradigms are similar enough. So I'm confident that you'll become productive in no time.

ZettelDistraction

Is the plugin system TypeScript-compatible? I guess the Javascript-compiled output would be...

ctietze

@ZettelDistraction The system doesn't perform any pre-processing step for you, only plain JavaScript (ES6, I believe? Need to check!). To get started, it all needs to be contained in 1 file, so for complex plugins, Babel or whatever needs to be used? Using require(...) to import files from the plugin bundle is on my list, but that's beyond a first version to do things with

If you're a TypeScript user, I'd eventually really like to explore how to make the workflow bearable. Delivering type annotations, for example.

ctietze

• The JavaScript code without syntax highlighting is, uh, a bit on the ugly and confusing side.
• Running the script with insufficient access permissions correctly produces an error (see the console). I then enabled the expected input ("All Notes") and output ("New File").
• The configuration inspector to the right is almost complete. That's all the inputs and outputs in their boring glory at the moment. But the file visiting one's are bound to be made user configurable instead of the script author's decision.

The script I executed is a built-in one: "Statistics of All Tags":

let tags = {};

// match[0] is the whole hashtag, match[1] the tag name without the hash
let tagRegex = /(?:\s|^)#(\w*[A-Za-z_]+\w*)/g;
for (let note of input.notes.all) {
    for (let tagMatch of note.content.matchAll(tagRegex)) {
        let tag = tagMatch[1].toString().toLowerCase();
        tags[tag] = (tags[tag] === undefined) ? 1 : tags[tag] + 1;
    }
}

// Object.entries produces [["tagname",123],...]
let sortedTagCountPairs = Object.entries(tags).sort((lhs, rhs) => lhs[1] < rhs[1]);
let body = "Tags:\n";
for (let tagCount of sortedTagCountPairs) {
    body += tagCount[1] + " #" + tagCount[0].toString() + "\n";
}
output.newFile.content = body;

It's done for quite some time (2022-08-06) and the plugin management window shows it's info:

ctietze

The "Copy Settings as JavaScript" button is meant to kick-start development with a template that exposes all the inputs and outputs.

The mechanisms are a bit simple for now. If you selected all the options, you'd get this

// Each note object has these properties:
// - `path`: The absolute POSIX path to that file.
// - `filename`: The filename without extension. 
//    (Can be considered the title.)
// - `content`: The full text of the note.

// Array of all notes in the user's archive.
const allNotes = input.notes.all;

// Array of selected notes. Can be empty, the currently visited (edited) note,
// or multiple notes for bulk operations.
const selectedNotes = input.notes.selected;

// Full text of the currently visited note.
// (Is the same as `selectedNotes[0].content` when a single note is selected.)
const editorContents = input.text.all;

// Selected text in the currently visited note. Can be empty
// when no note is visited or the selection is empty
// (that is, when the cursor blinks normally).
const selectedText = input.text.selected;

// 📝 Your Script Goes Here!

// Set the text to insert on behalf of the user
// in the editor when the script has finished.
// (Affects the currently visited file, not the one you'll be creating.)
output.insert = "";

// Set the contents of a new file that the app should
// create when the script has finished.
output.newFile.content = "";

ctietze

Writing JavaScript code without any visual guidance is quite the challenge, I found. I'm spoiled

Luckily, to get rudimentary syntax highlighting for a well-known language like JavaScript to work requires just the inclusion of a couple of simple open source libraries, and things are much more readable:

ZettelDistraction

Very nice.

EternalRecursion

I presume development of The Archive is in Swift using Xcode? (More of a question than an actual presumption lol.)

What open source libraries are you using to facilitate the JavaScript execution and editing features?

-- ER

ctietze

@EternalRecursion It is! JavaScriptCore is a first-party (read: Apple-made) library for this. It's basically the JavaScript engine built into browser via the WebKit engine by default. The plumbing to make The Archive accessible from JavaScript is custom.

romanthekat

Can't wait for it to happen I do have a few cases where extra custom logic would be handy

ctietze

Progress Report 2024-07-14

In my last update, I talked about the editor/development console. To rapidly iterate (and not compile for hours) I worked on that in a separate module. In December I began the integration work, then worked on other issues in parallel and cleaned up the text editor a bit, so it took the past ~2--3 days to integrate everything for real and resolve issues. (I'm not particularly good at interpreting the kind of errors you get there by build tools, but I'm much better at it than I was 1 or 2 years ago, thankfully.)

Evaluating the first script in 2024

With the dev tool available from the app, the result of my first script execution is this:

You see a selection of notes in The Archive at the top (pointing at my dev/test note collection where, apparently, I also debugged issues with image inclusion that @Will reported a while ago ).

The code in the dev tool is:

input.notes.selected.map((n) => n.filename)

This obtains all selected notes and from each note picks the filename. That's the highlighted text at the bottom: a JavaScript representation of an array of the filenames as strings.

The dev tool allows to experiment with things on the fly and remote-control The Archive to some extent. This example is just reading selected notes. So ...

Producing text on my behalf

A very simple and rather safe changes is to make a script compute 1 large text output and insert it on the user's behalf:

output.insert.text = "Hello, World!"

That will 'type' the text Hello, World! into the currently frontmost editor window where the cursor is.

Safety first

This kind of output is limited to 1 change per script run. That means consecutive setter statements will overwrite previous ones.

For example, this will only produce "Bye" after execution:

output.insert.text = "Hello"
output.insert.text = "Bye"

This is the current design for what'll become the lowest risk kind of plugins. Plugins that offer simple automations where the assembly of information can be as convoluted as you like, but the users only needs to understand whether the script will e.g. insert text once into the editor or create a new file.

That's fairly limiting for power users, so that's not all there is to it, but that's all I want to show today

ctietze

Two sample scripts for Zettelkasten auditing I work with since the beginning do what they are supposed to do from within The Archive:

• Find Orphaned Notes: lists all notes that have no incoming links
• Statistics of all Tags: lists all hashtags and how often they appear, sorted by number of uses

ctietze

Last week, I opened up how The Archive and plugins communicate a bit more to implement a script to perform the Extract Selection as New Note Zettel refactoring.

Going through a list of script ideas we've collected over the years showed that the app's capabilities weren't quite there, yet, to give users the power to implement this. While the plugin system is in place and runs fine now, the app feeding information into scripts/plugins and getting the result back sometimes reveals that I need to rethink (and replace) a feature that's been in the app for 6+ years.

Listing all notes, for example: In the app, once you start searching, the window shows matching notes in the left-hand sidebar. For all possible user interactions, this filtered list is all you need. No matter that you have 10,000 notes -- if you narrow it down to say 3 with a search, there's no affordance, no widget, to interact or see the remaining 9997. That's what search is for But plugins may need to filter through all notes, all 10k of them, not just the couple of search results. Since there was no need for the app's user interface to access all 10k notes, there actually was no way to provide these to plugins easily.

The Extract Note refactoring revealed how automation can perform actions that a user can't possibly do: change a note in the editor window and create a new file at the same time. A human user needs to click away from the editor or perform a shortcut. There's always some interaction that tells the app that the user has finished typing. Plugins don't click or perform key presses, so command sequences from natural input can be performed in a new order by the computer.

Observing how The Archive behaves with commands coming from a totally different angle produced surprising inconsistencies that I needed to address. "Humanly possible" is not the same as "possible" in this case. Addressing these new emergent behaviors will ultimately harden the app for human input, so that's good, and having a new way to modify notes in an automated fashion can help to automate testing the app, and thus prevent bugs in the future.

But touching years old subsystems that "just worked" is very dangerous. Unknown unknowns can lurk around any corner. That's always a weird moment in software development. So I'm running the app for myself with extra safety gloves (git version control with daily snapshots) at the moment to verify that at least I notice when something unexpected happens.

ctietze

.thearchiveplugin plugin bundles can now be opened in the app to install them. That makes sharing (and experimenting) easy enough for a release very soon.

ctietze

Alright, things seem to run smooth.

Brace yourselves to update to v1.8.0 and the new plugin feature later this month!

ctietze

Things work fine so far.

I'm still testing plug-in writing and organize ideas for scripts and workflows we collected over the years.

Documentation still needs to go online, too, to ease onboarding and playing with it.

If you know that you will want to write plug-ins, reach out via DM!

ctietze

Got invaluable feedback from @Will yesterday. Thank you!

With that, I can tweak things a bit to help pro-users and automation fans to start writing plugins. Some essential functionality and inspection tools are missing at the moment, which makes getting started tough.

Also there's no scripting documentation page, yet. I'm drafting a guide but haven't finished it, yet.

ctietze

I'm getting very useful feedback from a plug-in developer's perspective. Almost fixed all the things -- so that I can onboard the next bunch of testers later today or maybe tomorrow with a test build of The Archive.

ctietze

Update for everyone: we've been inviting The Archive customers to participate and the response was (for me) overwhelmingly positive. Around 75 people responded within days. Let's see how their testing goes!

We're collectively working on a vision to make this all better and more powerful.

I'm really grateful for all of this, thank you!

Also, the first release's set of features is solid and I'm fixing some remaining weirdnesses and bugs people discover. So v1.8.0 is near, then we'll expand later.

mediapathic

Fantastic to see the progress on plugins @ctietze ! I'm so glad you guys are still actively working on this. I just did my periodic "I'm tired of all this bullshit I just want my Archive back" as I do from time to time (also, actually, testing switching my sync over to iCloud so wanted to make sure my methods were compatible with the most important tools), and I'm pleased to swing by again in my asymptotic orbit and see things still going strong here. Much love to you folks as always.

ctietze

@mediapathic Happy to see you say hi every now and then!

Penultimate update?

Last week was crazy with the newborn: Death by 1000 cuts, too many interruptions made meaningful progress and thought impossible at times.

But I managed to draw the plug-in developer UI change for the next update before I could implement it:

And then I implemented the changes -- not just visually, but also a bit of new logic underneath:

Before	After

Will ship to beta testers ASAP (today or tomorrow).

That's going to introduce one last breaking change with some existing plug-ins, then it's time to massage a public release.

ctietze

Ready to ship in a couple of days

I believe I've tackled all issues in the actual app. I'm continuing to work on the docs to get people started now.