My additions are (obviously) one outer and inner square bracket, which I escaped appropriately. The syntax is now adding brackets and bookmarks as expected.
What you did is change the recognition of single bracket note links to double bracket note links. That would mean that single bracket note links won't work anymore, right?
To fix it for both you could change the first term to ([\]]?\[) and the last to (\][\]]?) but I haven't tried that. If that fixes it for you for both single and double bracket links, I should better incorporate that into the syntax file so everybody can benefit from it
Great stuff, thank you for sharing your issue! The community is going to benefit
@zverhope I tested the above on my work machine and it does not cause troubles (as expected). Because it seems to fix your issue I have commited it to github. It's now official
this plugin just gets better and better. sooner or later it will be so advanced that it will write the zettel for you, freeing up your time to spend on reading and thinking!
Iiiiintroducing... The latest awesome feature: "Finding Friends"
It's about finding notes also referencing another note: If you encounter a link to another note, click on it and press [alt]+[enter]. That will open the already-well-known find-in-files panel, pre-filled to search for references to the note-id in question. I call notes that link to the same note as the open one: friend notes of the open one, hence "finding friends"
Is there a reason you stop at 1 bracket and not support both? Matcher suggestion that might work if I got back-references right:
- match: '(\[{1,2})([0-9]{12})(\1)'
Using "The Silver Searcher"?
A beta tester of The Archive recently requested ag (aka "The Silver Searcher") search results. Since your tag list and search results seem to pop up in a new buffer/pane, this could be useful. You can use ag to show search results in a compact fashion, like:
FILENAME : LINE_NO : line from file with matched search result highlighted
It is fast enough for live-filtering a directory of notes with some delay.
Is there a reason you stop at 1 bracket and not support both?
I don't stop at one bracket: ([\]]?\[) does a very similar thing as (\[{1,2}). While you tell it to look for one or two brackets, I ask it to look for an optional bracket, followed by a bracket. So both support one or two brackets
Using "The Silver Searcher"?
Hmmm, now that looks interesting! Will definitely check it out!
Here's a bash script/command I wrote to generate tag lists from a folder of notes and populate an Atom editor view with the results. Might give you a starting point to work from. (the reason I had to make a tempfile and display that file in Atom is because Atom didn't support "scratch" buffers like Sublime does) -- let me know if the scripts have any errors, I had to pull them out of a plumbing file and I might have missed something in translation.
This command was triggered when a user followed the special #all tag-link which searches for all tags in all notes in the folder:
#!/bin/bash
dir=.
editor=subl
tf="$(mktemp /tmp/#all.XXX.md)"
echo -en "all # tags\\n" >> "$tf"
ag --markdown --nocolor --nofilename -o "(?<=^|(?<=[^.?\\/[:alnum:]]))#([\\w\\-]+)" "$dir" | sort -u | sed "/#all/d" | cat >> "$tf"
$editor "$tf"
unset dir
unset tf
unset editor
This command was triggered when a user followed a tag link in the format of #ALPHANUM_STRING:
#!/bin/bash
dir=.
editor=subl
tf="$(mktemp /tmp/#"$1".XXX.md)"
echo -en "notes tagged with #$1\\nshow #all\\n\\n" >> "$tf"
ag -lS "#$1" "$dir" | sed "s:$dir/::g" | sort | cat >> "$tf"
$editor "$tf"
unset dir
unset tf
unset editor
Also of note for you @rene is this package which allows you to use ag, grep, pt and other tools directly from within Sublime: https://packagecontrol.io/packages/Search in Project might be useful to look at their code, or even just offload the work to their package
Thanks for the suggestions regarding ag, grep, pt, etc.
I will think about it and probably work something out. The only reason I don't want to rely too heavily on those is that grep, bash & co are only available per sort-of default on unix-like systems. you can't expect to find them on Windows.
I see SublimeText as a possibility to have a good platform-agnostic tool for implementing the Zettelkasten method. Then tying it too closely to linux/OSX commands would defeat that purpose.
But to have those tools as options is probably a sensible idea. I do not plan, however, to replace the ('slow') python search by grep. That would make it too unusable on Windows.
I work exclusively on Linux and OS X at the moment but I have so many colleagues and friends who just only know Windows. They should not be left out because they don't know any better.
Good point, getting ag on Windows is not trivial. I can't give any advice on how to write a Sublime plugin. I guess providing a default plugin configuration value would help. Maybe search = "default" to use the Python default can work, then interpret anything != "default" as a shell command so that search = "grep -x -y -z" searches via grep.
I need some clarifications , maybe you can help me with your views.
And I want to provide some perspective.
as @toolboxen suggested, instead of ag one could use pt which seems to be even faster than ag (in some cases) but, more importantly to me, has pre-built binaries for all major platforms.
But why use it?
compact output: In SublimeText's find-in-files, you can toggle a button to show context lines or not. No context reduces the results to 2 lines: file name and matching line. OK, not perfect but better than 5 lines (matching line + 4 context lines).
SublimeText's find-in-files is really fast: In a test archive with over 10 000 notes, the find results showed up instantly.
ag and pt are fast, but:
they are fast because they utilize all cores -> machine becomes super busy for a short time
that dictates it is not wise to have it running too frequently for live-updates
but, to put that in context: in an archive with 10 000 notes it would probably finish within milliseconds, so, theoretically it could be called every 1.5 seconds to provide a self-updating tag list of sort.
So, to me, find-in-files is still a good compromise: it's lightning fast and needs no external dependencies.
For the tag list, however, I search in python. That is bound to be much slower than all the other methods. However, I have tested it on my admittedly fairly powerful Macbook Pro: I made copies of my notes so that in total I have 10008 notes in the test archive and #! produced a note list in 1.3 seconds.
Again, to me 1.3 seconds in a hideously large Zettelkasten archive is a good compromise. Live-update is out of question, though. Well, I could put it into a background thread and refresh every n seconds but I don't like the idea of creating such a kind of permanent workload just for a self-updating tag list that doesn't change that frequently.
So what do I make of this? Hmmm, providing a configuration setting: path_to_pt for advanced users is an option. If it points to an existing executable, it would be used to create the tag list and have the tag-list self-update, too.
Potentially it could be used to produce a compact, custom find results view, but: that iiiis a bit more involved because then I'd have to implement my own search results view. The 'Search in projects' plugin mentioned earlier in this thread provides an overlay which I think is not the best way to do it. It additionally produces a "Find results" buffer that looks like SublimeText's find results but it has one major drawback: You can't click on a result to open the specific file. That makes it pretty unusable in my eyes. So that would need to be fixed. I am not 100% sure whether SublimeText supports a way for click-to-open in plugins. BetterFindBuffer (https://github.com/aziz/BetterFindBuffer) gets around that by creating a read-only view and using keyboard shortcuts like 'o' or 'enter' to 'open', etc.
#? to insert a tag from a fuzzy-searchable list of tags needs a speed-up though. waiting for 1-2 seconds in a 10 000 notes archive just to insert a tag defeats the purpose of that feature. But this could be solved without resorting to external dependencies by caching #tags in some buffer / file and updating it in the background every 5-10 seconds or so.
[alt] + double clicking a note-link will open the corresponding note
[alt] + double clicking a tag will open the tag search
Experimenting with the mouse was also a bit of pre-work for later, to make use of pt for searching, having a custom find results view with clickable search results
@toolboxen: thanks again for convincing me to use a custom syntax / scope. That makes things like mouse support really easy now
I see SublimeText as a possibility to have a good platform-agnostic tool for implementing the Zettelkasten method. Then tying it too closely to linux/OSX commands would defeat that purpose.
Agreed! Keep it cross platform. Use python to script and cross-platform binaries to do the heavy lifting. (or just rely on python for everything if you can develop a performant approach) BTW, the most important part of the two scripts I shared are the ag lines, after some testing it was those ag commands that generated the best tag lists. pt accepts most if not all the same flags as ag.
But to have those tools as options is probably a sensible idea.
Yeah, perhaps if available the plugin can use them, if not default back to python searching.
They [Windows users] should not be left out because they don't know any better.
Hey now! I'm using Windows! In the past I've used OSX and Linux with equal intensity. Each has pros/cons. Today cross-platform support is better than ever, and I've found there's usually a way to get the job done on any platform.
instad of ag one could use pt which seems to be even faster than ag (in some cases) but, more importantly to me, has pre-built binaries for all major platforms.
ag and pt are one-line installs on OSX and Linux. There's a handy package manager for Windows: https://chocolatey.org to provide one-line installs of all sorts of nice packages like ag and pt -- these installs add the binaries to the Windows path.
SublimeText's find-in-files is really fast: In a test archive with over 10 000 notes, the find results showed up instantly.
Yeah, that search index which powers Gotoanything/FindInFiles is a strong point with Sublime, too bad there is no API to query against it! If there was, WOW all this work would be reduced to a few querys and displaying the results in overlays and scratch buffers.
they are fast because they utilize all cores -> machine becomes super busy for a short time
I think there are options to throttle? But i'm not sure.
that dictates it is not wise to have it running too frequently for live-updates... in an archive with 10 000 notes it would probably finish within milliseconds ... theoretically it could be called every 1.5 seconds
When I was testing ag and pt on 10000 markdown file I decided against doing background updates instead opting to run a new search each time. The live search was faster than my terminal could update. I was pushing the search results over into an Atom buffer. It took Atom almost 10 times longer to display an open file buffer than it did to run the search! I reasoned that we since we have multiple cores waiting to be used, let's use them! Give me an instant search result every time I ask for it. Just in time searching...
For the tag list, however, I search in python. That is bound to be much slower than all the other methods. However, I have tested it on my admittedly fairly powerful Macbook Pro: I made copies of my notes so that in total I have 10008 notes in the test archive and #! produced a note list in 1.3 seconds.
In my tests both pt and ag were almost 10x that speed on 10000 markdown files.
Well, I could put it into a background thread and refresh every n seconds but I don't like the idea of creating such a kind of permanent workload just for a self-updating tag list that doesn't change that frequently.
I agree, in theory it's not necessary. I prefer: Get the fastest search tool you can, and generate a new tag list each time. However on older systems this might actually be perceptually slower than background python/ag/pt generating a scratch file every 3/5/10 seconds.
configuration setting... pointing to an existing executable, it would be used to create the tag list and have the tag-list self-update, too.
Or just test the Path environment variable for pt or ag and if available use them if not fall back to python. Reduce the configuration variable complexity wherever possible. New users 1. install sublime 2. install sublime_zk 3. optional: install ag or pt -- boom! you're done.
produces a "Find results" buffer that looks like SublimeText's find results but it has one major drawback: You can't click on a result to open the specific file....I am not 100% sure whether SublimeText supports a way for click-to-open in plugins.
Sublime has mouse event listeners and you should be able to generate some scopes in a search results buffer to enable clickable links
#? to insert a tag from a fuzzy-searchable list of tags needs a speed-up though. waiting for 1-2 seconds in a 10 000 notes archive just to insert a tag defeats the purpose of that feature.
Install and test https://packagecontrol.io/packages/Search in Project and see how fast it runs on 10000 files using ag or pt. If opening an overlay has no discernible delay then you could use that approach for your for fuzzy searchable list of tags.
But this could be solved without resorting to external dependencies by caching #tags in some buffer / file and updating it in the background every 5-10 seconds or so.
That might be a better option than live searching especially on slower systems. In the past I considered saving ag searches to temp files in a .tags folder and then opening those tag files in Atom or Sublime to interact with the tag lists. Both editors offer live reloading upon file changes, so the tag list would be always up to date!
@toolboxen: thanks again for convincing me to use a custom syntax / scope. That makes things like mouse support really easy now
You're welcome! it's cool to see how quickly you get things working! I am very hopeful in future that Sublime will open up the search index with an API. It will make all of this super easy.
I was curious so after doing some testing with pt and ag on Win8.1 on my old Thinkpad T400 (core 2 duo 2.4ghz p8600), it seems pt is faster than ag searching 10k markdown files for short words like test and the but it is taking a lot longer than when I tested using my I7 quad-core
ag the > search1.md: ~9-10 seconds ag test > search2.md: 4 seconds pt the > search3.md: ~2-3 seconds pt test > search4.md 1 second
fwiw, these are run in cmd.exe and not Powershell.
@toolboxen, thanks a lot for your elaborate answer!
I'll try to make it short: Initial support for The Silver Searcher is here
Currently it speeds up the search for all tags ([#][?] tag selector and [#][!] tag list in new tab). If it's in the path, it's detected automatically, so no need to configure a thing (@toolboxen: great tip ).
Why ag and not pt? Though pt would be easier to install on all platforms, ag has one feature that pt doesn't: --only-matching. That allows for a tag search that results only in tags instead of complete lines containing the tags.
So no parsing of the output is required to build the tag list. This is relevant in large archives where every note contains at least one line with tags.
I would like to add more ag to the plugin, not for performance reasons but for convenience: bringing up a search-in-files panel (as it is done now to search for tagged notes and friend notes) requires the user to press [enter] which seems a bit odd.
ag now also speeds up searching for friend notes ([alt]+[enter] on a note-link shows a list of notes that also link to the same note) and tagged notes ([ctrl]+[enter] or [alt]+click on a #tag 'follows' the tag by showing a list of notes tagged by the #tag).
I went for the easy way: showing an overlay (the typical SublimeText list that can be narrowed down by typing).
So no more pre-filled find-in-files panels anymore.
If ag is not found on the system, default implementations
being slow (producing a tag list / tag selection overlay)
or
bringing up a find-in-files panel (tagged / friend note list)
Forgot to mention: A find-results tab is cool because it can provide context, i.e. the line containing a tag and optional additional context lines.
However, I am quite pleased with how it turned out. The overlays go away when they are not used anymore...
But when I think of it, there might be use-cases when one wants a permanent search result. I am thinking of commands that produce note links with descriptions based on search results:
If you want to compile a list of all notes tagged with #tag or referencing [[note-id]], it would produce a buffer like this:
<!-- Notes containing ##silver-searcher -->
[[note-id 01]] Title of note one
[[note-id 02]] Title of note two
[[note-id-37]] Tite of that note
or similarly
<!-- Notes referencing [[note-id 100]] -->
[[note-id 01]] Title of note one
[[note-id 02]] Title of note two
[[note-id-37]] Tite of that note
So you get ready-to-copy snippets for pasting into some overview note or as a start for an outline...
@rene said
Why ag and not pt? Though pt would be easier to install on all platforms, ag has one feature that pt doesn't: --only-matching. That allows for a tag search that results only in tags instead of complete lines containing the tags.
Yeah that's the same reason why I stuck with ag as well. If pt ever adds that option you should consider switching as the speed advantage is worth it.
I'll add a pull request to your readme for instructions on installing ag in windows.
The overlays go away when they are not used anymore... But when I think of it, there might be use-cases when one wants a permanent search result.
I am thinking of commands that produce note links with descriptions based on search results...
Nice! I was going to mention this a something to consider. You beat me to it. Both overlays and note lists are useful. Having these lists is handy you can go through them and follow specific links to certain notes or like you said copy the entire list to create an overview note.
fwiw, here's how I generate my tag headers, I include a link to the #all tag which is a special tag for me, it shows all tags. I don't bother with html comments <!-- -->
notes tagged with ##silver-searcher
show #all
[[note-id 01]] Title of note one
[[note-id 02]] Title of note two
[[note-id-37]] Tite of that note
clicking on the #all tag generates this list for me:
all # tags
##silver-searcher
#toolboxenrocks
#reneisawizard
#taggytag
Clicking on any one of those tags leads back to a list of notes containing that tag. So I zoom between buffers and they form a text-based UI generated on the fly from the contents of the notes.
when I've copied markdown links from twitter users who ended their tweets with hashtags.
2) I've got a lot of tags, and thus a lot of noise in my tags list. A bunch of those are edge cases I don't see any way of getting around, like however I wound up with #I'm. But being able to have an option to ignore numbers as well would significantly reduce that noise for me.
3) I have done the brew ag install, restarted Sublime, and am still getting default find windows. Is there something else I need to do to make Sublime talk to ag? Clarifying this in your readme would probably be helpful as well.
[edit] (while I'm here, while this is not your problem, does anyone know how to follow a link in the generated file result list with a keyboard shortcut and not double clicking?)
Changing the ag regex to (?<=^|(?<=[^.?\/[:alnum:]]))#([\w\-]+) will solve a lot of these problems. That is the regex I came up with for my tag system. Here's a test demonstrating that it solves half the problems presented by @mediapathic
As you can see it still fails on the #I'm and possibly on the twitter link.
@mediapathic where and why does the #I'm appear in your text? is that a markdown heading?
[edit] (while I'm here, while this is not your problem, does anyone know how to follow a link in the generated file result list with a keyboard shortcut and not double clicking?)
I once tested the regexp above with grep and had issues. I think it was with rather simple things like ##AI or sth. But OK, the regexes need to be changed, I agree
@mediapathic
regarding brew: Bring up the console window, if execution of ag fails it should provide diagnostic information
I once tested the regexp above with grep and had issues. I think it was with rather simple things like ##AI or sth. But OK, the regexes need to be changed, I agree
Sure cause I don't use double hashtags so I only grepped for single tags.
@mediapathic where and why does the #I'm appear in your text? is that a markdown heading?
No, it's a hashtag referenced in an article I quote in a document, I just mentioned it as an example of noise. That firmly falls under not your problem to solve
[edit] (while I'm here, while this is not your problem, does anyone know how to follow a link in the generated file result list with a keyboard shortcut and not double clicking?)
Should be [ctrl]+[enter] same as note links.
That's what I was afraid of. That's not working for me. I'm assuming this is a problem not related to your plugin though.
@toolboxen yeah, sure, we'll work it out. am a bit stressed at work but will check it out later.
@mediapathic correct, I meant the SublimeText console. If you see nothing there then the plugin has not attempted to do anything, at least not with ag. Because on error it will print out some diagnostics, like command issued, return code, and output. On timeout it will also print a message. I will put some diagnostics in indicating whether ag was found or not. Are you sure, it is installed in your PATH? That is the only requirement for the plugin. A command named ag must be executable without specifying it by its full path. I have tested it on OS X and now also on Linux. If you bring up a new system console and type ag [enter] and that works, then the plugin should work.
I'll post here when I have put in a startup message of the plugin (in the sublime console) that tells whether ag was found or not.
Regarding permanent search results: I think that'd be super useful! Picking up the tip by @toolboxen from above about hidden temporary files, why not route the filtered search results to a .search.txt file and display in a separate pane? With clickable [[...]] links, you end up with a clickable list of search results much like nvALT/The Archive.
It grew over time as double-hashtags and some special characters, chinese characters and stuff like that became relevant. Also tries to not match hashtags in code markup with (?<!\)`. Might not be the best implementation, yet.
Comments
@zverhope awesome!
What you did is change the recognition of single bracket note links to double bracket note links. That would mean that single bracket note links won't work anymore, right?
To fix it for both you could change the first term to
([\]]?\[)and the last to(\][\]]?)but I haven't tried that. If that fixes it for you for both single and double bracket links, I should better incorporate that into the syntax file so everybody can benefit from itGreat stuff, thank you for sharing your issue! The community is going to benefit
Just in case my edits got lost:
([\]]?\[)and(\][\]]?)@zverhope I tested the above on my work machine and it does not cause troubles (as expected). Because it seems to fix your issue I have commited it to github. It's now official
I just grabbed the package upgrade. As you suggested, both single and double brackets are working perfectly. So, so pleased!
Let me know if I can buy you a coffee or beer. Thanks again!
Awesome!!! Happy that it works for you now! Thanks for your contribution! (No donations required but thanks a lot for the gesture
)
this plugin just gets better and better. sooner or later it will be so advanced that it will write the zettel for you, freeing up your time to spend on reading and thinking!
Iiiiintroducing... The latest awesome feature: "Finding Friends"
It's about finding notes also referencing another note: If you encounter a link to another note, click on it and press
[alt]+[enter]. That will open the already-well-known find-in-files panel, pre-filled to search for references to the note-id in question. I call notes that link to the same note as the open one: friend notes of the open one, hence "finding friends"Here's what it looks like (this time on Linux):
... pressing
[enter]...Ta-dah
Is there a reason you stop at 1 bracket and not support both? Matcher suggestion that might work if I got back-references right:
- match: '(\[{1,2})([0-9]{12})(\1)'Using "The Silver Searcher"?
A beta tester of The Archive recently requested
ag(aka "The Silver Searcher") search results. Since your tag list and search results seem to pop up in a new buffer/pane, this could be useful. You can useagto show search results in a compact fashion, like:It is fast enough for live-filtering a directory of notes with some delay.
Author at Zettelkasten.de • https://christiantietze.de/
I don't stop at one bracket:
([\]]?\[)does a very similar thing as(\[{1,2}). While you tell it to look for one or two brackets, I ask it to look for an optional bracket, followed by a bracket. So both support one or two bracketsHmmm, now that looks interesting! Will definitely check it out!
Here's a bash script/command I wrote to generate tag lists from a folder of notes and populate an Atom editor view with the results. Might give you a starting point to work from. (the reason I had to make a tempfile and display that file in Atom is because Atom didn't support "scratch" buffers like Sublime does) -- let me know if the scripts have any errors, I had to pull them out of a plumbing file and I might have missed something in translation.
This command was triggered when a user followed the special
#alltag-link which searches for all tags in all notes in the folder:This command was triggered when a user followed a tag link in the format of
#ALPHANUM_STRING:pt has better cross platform support than ag but doesn't offer exactly the same command structure: https://github.com/monochromegane/the_platinum_searcher
Also of note for you @rene is this package which allows you to use ag, grep, pt and other tools directly from within Sublime: https://packagecontrol.io/packages/Search in Project might be useful to look at their code, or even just offload the work to their package
Thanks for the suggestions regarding ag, grep, pt, etc.
I will think about it and probably work something out. The only reason I don't want to rely too heavily on those is that grep, bash & co are only available per sort-of default on unix-like systems. you can't expect to find them on Windows.
I see SublimeText as a possibility to have a good platform-agnostic tool for implementing the Zettelkasten method. Then tying it too closely to linux/OSX commands would defeat that purpose.
But to have those tools as options is probably a sensible idea. I do not plan, however, to replace the ('slow') python search by grep. That would make it too unusable on Windows.
I work exclusively on Linux and OS X at the moment but I have so many colleagues and friends who just only know Windows. They should not be left out because they don't know any better.
Good point, getting
agon Windows is not trivial. I can't give any advice on how to write a Sublime plugin. I guess providing a default plugin configuration value would help. Maybesearch = "default"to use the Python default can work, then interpret anything!= "default"as a shell command so thatsearch = "grep -x -y -z"searches via grep.Author at Zettelkasten.de • https://christiantietze.de/
I need some clarifications
, maybe you can help me with your views.
And I want to provide some perspective.
as @toolboxen suggested, instead of ag one could use pt which seems to be even faster than ag (in some cases) but, more importantly to me, has pre-built binaries for all major platforms.
But why use it?
compact output: In SublimeText's find-in-files, you can toggle a button to show context lines or not. No context reduces the results to 2 lines: file name and matching line. OK, not perfect but better than 5 lines (matching line + 4 context lines).
SublimeText's find-in-files is really fast: In a test archive with over 10 000 notes, the find results showed up instantly.
ag and pt are fast, but:
that dictates it is not wise to have it running too frequently for live-updates
So, to me, find-in-files is still a good compromise: it's lightning fast and needs no external dependencies.
For the tag list, however, I search in python. That is bound to be much slower than all the other methods. However, I have tested it on my admittedly fairly powerful Macbook Pro: I made copies of my notes so that in total I have 10008 notes in the test archive and
#!produced a note list in 1.3 seconds.Again, to me 1.3 seconds in a hideously large Zettelkasten archive is a good compromise. Live-update is out of question, though. Well, I could put it into a background thread and refresh every n seconds but I don't like the idea of creating such a kind of permanent workload just for a self-updating tag list that doesn't change that frequently.
So what do I make of this? Hmmm, providing a configuration setting:
path_to_ptfor advanced users is an option. If it points to an existing executable, it would be used to create the tag list and have the tag-list self-update, too.Potentially it could be used to produce a compact, custom find results view, but: that iiiis a bit more involved because then I'd have to implement my own search results view. The 'Search in projects' plugin mentioned earlier in this thread provides an overlay which I think is not the best way to do it. It additionally produces a "Find results" buffer that looks like SublimeText's find results but it has one major drawback: You can't click on a result to open the specific file. That makes it pretty unusable in my eyes. So that would need to be fixed. I am not 100% sure whether SublimeText supports a way for click-to-open in plugins. BetterFindBuffer (https://github.com/aziz/BetterFindBuffer) gets around that by creating a read-only view and using keyboard shortcuts like 'o' or 'enter' to 'open', etc.
These are just my thoughts... What do you tink?
#?to insert a tag from a fuzzy-searchable list of tags needs a speed-up though.New feature: mouse support:
[alt]+ double clicking a note-link will open the corresponding note[alt]+ double clicking a tag will open the tag searchExperimenting with the mouse was also a bit of pre-work for later, to make use of pt for searching, having a custom find results view with clickable search results
@toolboxen: thanks again for convincing me to use a custom syntax / scope. That makes things like mouse support really easy now
Agreed! Keep it cross platform. Use python to script and cross-platform binaries to do the heavy lifting. (or just rely on python for everything if you can develop a performant approach) BTW, the most important part of the two scripts I shared are the ag lines, after some testing it was those ag commands that generated the best tag lists. pt accepts most if not all the same flags as ag.
Yeah, perhaps if available the plugin can use them, if not default back to python searching.
Hey now! I'm using Windows! In the past I've used OSX and Linux with equal intensity. Each has pros/cons. Today cross-platform support is better than ever, and I've found there's usually a way to get the job done on any platform.
ag and pt are one-line installs on OSX and Linux. There's a handy package manager for Windows: https://chocolatey.org to provide one-line installs of all sorts of nice packages like ag and pt -- these installs add the binaries to the Windows path.
Yeah, that search index which powers Gotoanything/FindInFiles is a strong point with Sublime, too bad there is no API to query against it! If there was, WOW all this work would be reduced to a few querys and displaying the results in overlays and scratch buffers.
I think there are options to throttle? But i'm not sure.
When I was testing ag and pt on 10000 markdown file I decided against doing background updates instead opting to run a new search each time. The live search was faster than my terminal could update. I was pushing the search results over into an Atom buffer. It took Atom almost 10 times longer to display an open file buffer than it did to run the search! I reasoned that we since we have multiple cores waiting to be used, let's use them! Give me an instant search result every time I ask for it. Just in time searching...
In my tests both pt and ag were almost 10x that speed on 10000 markdown files.
I agree, in theory it's not necessary. I prefer: Get the fastest search tool you can, and generate a new tag list each time. However on older systems this might actually be perceptually slower than background python/ag/pt generating a scratch file every 3/5/10 seconds.
Or just test the Path environment variable for pt or ag and if available use them if not fall back to python. Reduce the configuration variable complexity wherever possible. New users 1. install sublime 2. install sublime_zk 3. optional: install ag or pt -- boom! you're done.
Sublime has mouse event listeners and you should be able to generate some scopes in a search results buffer to enable clickable links
Install and test https://packagecontrol.io/packages/Search in Project and see how fast it runs on 10000 files using ag or pt. If opening an overlay has no discernible delay then you could use that approach for your for fuzzy searchable list of tags.
That might be a better option than live searching especially on slower systems. In the past I considered saving ag searches to temp files in a
.tagsfolder and then opening those tag files in Atom or Sublime to interact with the tag lists. Both editors offer live reloading upon file changes, so the tag list would be always up to date!You're welcome! it's cool to see how quickly you get things working! I am very hopeful in future that Sublime will open up the search index with an API. It will make all of this super easy.
I was curious so after doing some testing with pt and ag on Win8.1 on my old Thinkpad T400 (core 2 duo 2.4ghz p8600), it seems pt is faster than ag searching 10k markdown files for short words like
testandthebut it is taking a lot longer than when I tested using my I7 quad-coreag the > search1.md: ~9-10 secondsag test > search2.md: 4 secondspt the > search3.md: ~2-3 secondspt test > search4.md1 secondfwiw, these are run in cmd.exe and not Powershell.
@toolboxen, thanks a lot for your elaborate answer!
I'll try to make it short: Initial support for The Silver Searcher is here
Currently it speeds up the search for all tags (
).
[#][?]tag selector and[#][!]tag list in new tab). If it's in the path, it's detected automatically, so no need to configure a thing (@toolboxen: great tipWhy ag and not pt? Though pt would be easier to install on all platforms, ag has one feature that pt doesn't:
--only-matching. That allows for a tag search that results only in tags instead of complete lines containing the tags.So no parsing of the output is required to build the tag list. This is relevant in large archives where every note contains at least one line with tags.
I would like to add more
agto the plugin, not for performance reasons but for convenience: bringing up a search-in-files panel (as it is done now to search for tagged notes and friend notes) requires the user to press[enter]which seems a bit odd.Anyone totally against continuing with
ag?Wowser, that was quick
agnow also speeds up searching for friend notes ([alt]+[enter]on a note-link shows a list of notes that also link to the same note) and tagged notes ([ctrl]+[enter]or[alt]+click on a #tag 'follows' the tag by showing a list of notes tagged by the #tag).I went for the easy way: showing an overlay (the typical SublimeText list that can be narrowed down by typing).
So no more pre-filled find-in-files panels anymore.
If
agis not found on the system, default implementationsor
will be used instead.
ta-dah
Forgot to mention: A find-results tab is cool because it can provide context, i.e. the line containing a tag and optional additional context lines.
However, I am quite pleased with how it turned out. The overlays go away when they are not used anymore...
But when I think of it, there might be use-cases when one wants a permanent search result. I am thinking of commands that produce note links with descriptions based on search results:
If you want to compile a list of all notes tagged with #tag or referencing [[note-id]], it would produce a buffer like this:
or similarly
So you get ready-to-copy snippets for pasting into some overview note or as a start for an outline...
Yeah that's the same reason why I stuck with ag as well. If pt ever adds that option you should consider switching as the speed advantage is worth it.
I'll add a pull request to your readme for instructions on installing ag in windows.
Nice! I was going to mention this a something to consider. You beat me to it. Both overlays and note lists are useful. Having these lists is handy you can go through them and follow specific links to certain notes or like you said copy the entire list to create an overview note.
fwiw, here's how I generate my tag headers, I include a link to the
#alltag which is a special tag for me, it shows all tags. I don't bother with html comments<!-- -->clicking on the #all tag generates this list for me:
Clicking on any one of those tags leads back to a list of notes containing that tag. So I zoom between buffers and they form a text-based UI generated on the fly from the contents of the notes.
Hey @rene this is great! Thanks so much for your work on this! Loving it so far!
A few small issues I'm having:
1) the tags list thinks punctuation is important. Therefore, my tags list looks like:
etc. This would also get rid of things like
#querytip"](https://twitter.com...when I've copied markdown links from twitter users who ended their tweets with hashtags.
2) I've got a lot of tags, and thus a lot of noise in my tags list. A bunch of those are edge cases I don't see any way of getting around, like however I wound up with
#I'm. But being able to have an option to ignore numbers as well would significantly reduce that noise for me.3) I have done the brew ag install, restarted Sublime, and am still getting default find windows. Is there something else I need to do to make Sublime talk to ag? Clarifying this in your readme would probably be helpful as well.
[edit] (while I'm here, while this is not your problem, does anyone know how to follow a link in the generated file result list with a keyboard shortcut and not double clicking?)
Thanks!
Changing the ag regex to
(?<=^|(?<=[^.?\/[:alnum:]]))#([\w\-]+)will solve a lot of these problems. That is the regex I came up with for my tag system. Here's a test demonstrating that it solves half the problems presented by @mediapathichttps://regex101.com/r/vf5dSY/2
As you can see it still fails on the
#I'mand possibly on the twitter link.@mediapathic where and why does the
#I'mappear in your text? is that a markdown heading?Should be
[ctrl]+[enter]same as note links.@toolboxen
I once tested the regexp above with grep and had issues. I think it was with rather simple things like
##AIor sth. But OK, the regexes need to be changed, I agree@mediapathic
regarding brew: Bring up the console window, if execution of ag fails it should provide diagnostic information
Sure cause I don't use double hashtags
so I only grepped for single tags.
EDIT for @rene
Fixed for multiple poundsigns but I'm sure it could be more improved:
https://regex101.com/r/vf5dSY/3
No, it's a hashtag referenced in an article I quote in a document, I just mentioned it as an example of noise. That firmly falls under not your problem to solve
That's what I was afraid of. That's not working for me. I'm assuming this is a problem not related to your plugin though.
I'm not sure if you mean the sublime console or the system console, but I don't see anything relevant on either.
@toolboxen
yeah, sure, we'll work it out. am a bit stressed at work but will check it out later.
@mediapathic correct, I meant the SublimeText console. If you see nothing there then the plugin has not attempted to do anything, at least not with ag. Because on error it will print out some diagnostics, like command issued, return code, and output. On timeout it will also print a message. I will put some diagnostics in indicating whether ag was found or not. Are you sure, it is installed in your PATH? That is the only requirement for the plugin. A command named
agmust be executable without specifying it by its full path. I have tested it on OS X and now also on Linux. If you bring up a new system console and typeag [enter]and that works, then the plugin should work.I'll post here when I have put in a startup message of the plugin (in the sublime console) that tells whether ag was found or not.
Regarding permanent search results: I think that'd be super useful! Picking up the tip by @toolboxen from above about hidden temporary files, why not route the filtered search results to a
.search.txtfile and display in a separate pane? With clickable[[...]]links, you end up with a clickable list of search results much like nvALT/The Archive.The official The Archive hashtag regex
So here's my trade secret: the hashtag regex
(?<=\s|^|\W)(?<!`)(#+[\p{L}\p{Nd}_§]*[\p{L}\p{Nd}_\-§]+[\p{L}\p{Nd}_§]*)It grew over time as double-hashtags and some special characters, chinese characters and stuff like that became relevant. Also tries to not match hashtags in code markup with
(?<!\)`. Might not be the best implementation, yet.Author at Zettelkasten.de • https://christiantietze.de/