I think the issue is not with deft per se, but with emacs. I hear complaints about slowness from a bunch of people on /r/emacs who are doing unrelated things, it seems to be just the case due to the internal structure of emacs that lots of files bogs it down. I think in theory the author of deft might be able to make structural changes about caching or whatnot, but I am not convinced even this will help. As far as I can tell the only real solution here is one of the projects trying to rewrite core emacs so that it is better at concurrency, but who knows when or if any of those might bear fruit.
This said, I found the thing that irritated me the most about slowness in deft was the autosave happening too frequently. I increase the time on that to I think a minute and it helped. Also maybe cleanup your buffer list more frequently, that was a big one for me. I realize these are not actually solutions to what you're asking, but they might help.
Do I get this right? Looking at deft.el, it looks like it's getting a list of files and then looks through these in deft-find-files.
This could be sped up considerably if one uses an on-disk search like ag/thesilversearcher. That's what @rene is doing in Sublimeless_ZK to great success. Not surprisingly, Emacs seems to sport compatible packages: https://cestlaz.github.io/posts/using-emacs-48-silversearcher/
My Elisp is ... well, let's say I cannot write an ag-based Deft that depends on file system data instead of an internal buffer cache. (That's what it seems to do, or else it wouldn't slow down Emacs completely, would it?)
@pinkhurricanegown I don't have that many notes in deft, but could you elaborate on precisely when you experience slowness? When using its interface to filter out notes?
Could you describe if that still happens when you use deft-filter to search rather than the usual deft interface?
@EFLS I can speak to my own experience here, which is, yes, using the interface to filter notes is desperately slow (~900 notes for me). Initializing deft takes about 8 seconds. The first letter I type into the filter interface takes about 4 seconds to show up, and after that of course it speeds up dependent on how many results there are. Using deft-filter search is perfectly reasonable.
I think I'm about where @mediapathic is. I've got 880 zettels. Initializing deft doesn't take super long for me, for whatever reason (~2-3 seconds) -- but searching is just painful. It's not only the number of Zettels, though -- I'm like 80% sure it's down to the way the searching works in deft.
Every time you add or remove a character to your serach, the whole thing is locking briefly (or not so briefly) while it does the incrementing/decrementing. When I run profiler and just do random searches, the report comes back with >45% of both the CPU and Memory time dedicated just to deft-filter-increment with something like 30% to deft-filter-decrement/deft-filter-decrement-word.
By contrast, if you use C-c C-l (by default) to call deft-filter, you can do enter-your-search-hit-return-get-results searching, which removes all the slowdown -- or, at the very least, hides it behind an acceptable pause while it does its thing.
From a user experience perspective, having a pause before displaying search results is vastly preferable to having input lag with every single keypress.
If I knew Elisp, I think the first project I'd take on is dismantling Deft to figure figure out how to add the option to have the search bar default to type-enter-results style searching, rather than incremental searching. Just adding that one option would fix most of my problems with Deft.
Either that, or I guess move to some other indexing back-end -- right now, Deft appears to just cat all the files in the directory into one giant string, and then searches that string for what you want.
The weird thing is, Notational Velocity/nvALT for example doesn't do anything more sophisticated than that: get the notes into memory and perform fast character-based search. Is it because there's no way to load stuff into Emacs memory without putting it into a buffer, and does that incur additional overhead? I don't know. But it's weird that this becomes an issue on modern day computing hardware even with an application core as old as Emacs's.
Totally! I used NVAlt for ages and ages, and never had this problem, even with a collection of notes twice what I've got now. It shouldn't be like this!
What makes it even more frustrating is, I know it's a problem with the Deft package and not Emacs itself.
Originally, all my zettels were individual headlines in one long Org file. I broke them out into individual files to move to Deft/Zetteldeft. When I re-combine all my current zettels back into an org file, it's only about 800kb -- we're not talking about a lot of data here, all told. Emacs can handle WAY bigger files like a champ.
When I dig around in that combined notes file using basic Emacs, there's no slowdown. Emacs only bites the dust when you're dealing with truly huge files, files with very long lines in them (1000s of characters long), or when you're applying a lot of custom font/color stuff with your theme. Or, when you're on Windows, there's an I/O bottleneck that limits the speed of certain kinds of processes.
And when I use other tools, there's no slowdown.
Take Swiper, for instance, which is a generic incremental search system that replaces the built-in search functionality across all of Emacs -- there's not the same lag when digging around in my zettels when they're one big file.
There's also org-velocity, which comes with the full install of Org and replicates the Notational Velocity experience, incremental searching and all, without slowdown.
So, why not use org-velocity or giant-file-with-Swiper? Because they're mediocre solutions. Deft/Zetteldeft is darn near perfect, except for this specific problem. And there aren't any better solutions for ZK on Windows -- much less solutions that will work across the Windows, macOS, and Linux machines I have to wander between for work.
Knowing what needs to be fixed, but not being able to fix it, is like having an itch somewhere that you can't reach -- maddening!
@pinkhurricanegown said:
Totally! I used NVAlt for ages and ages, and never had this problem, even with a collection of notes twice what I've got now. It shouldn't be like this!
What makes it even more frustrating is, I know it's a problem with the Deft package and not Emacs itself.
Originally, all my zettels were individual headlines in one long Org file. I broke them out into individual files to move to Deft/Zetteldeft. When I re-combine all my current zettels back into an org file, it's only about 800kb -- we're not talking about a lot of data here, all told. Emacs can handle WAY bigger files like a champ.
When I dig around in that combined notes file using basic Emacs, there's no slowdown. Emacs only bites the dust when you're dealing with truly huge files, files with very long lines in them (1000s of characters long), or when you're applying a lot of custom font/color stuff with your theme. Or, when you're on Windows, there's an I/O bottleneck that limits the speed of certain kinds of processes.
And when I use other tools, there's no slowdown.
Take Swiper, for instance, which is a generic incremental search system that replaces the built-in search functionality across all of Emacs -- there's not the same lag when digging around in my zettels when they're one big file.
There's also org-velocity, which comes with the full install of Org and replicates the Notational Velocity experience, incremental searching and all, without slowdown.
So, why not use org-velocity or giant-file-with-Swiper? Because they're mediocre solutions. Deft/Zetteldeft is darn near perfect, except for this specific problem. And there aren't any better solutions for ZK on Windows -- much less solutions that will work across the Windows, macOS, and Linux machines I have to wander between for work.
Knowing what needs to be fixed, but not being able to fix it, is like having an itch somewhere that you can't reach -- maddening!
What I really want here is deft to have some kind of option to use whatever search strategy works for the user. So, default would be native deft search, but I could set a flag and use ag, someone who likes xapian could use that, etc. I have no idea how difficult that would be to implement, so, no, I'm not volunteering to write it
Comments
@pinkhurricanegown
I think the issue is not with deft per se, but with emacs. I hear complaints about slowness from a bunch of people on /r/emacs who are doing unrelated things, it seems to be just the case due to the internal structure of emacs that lots of files bogs it down. I think in theory the author of deft might be able to make structural changes about caching or whatnot, but I am not convinced even this will help. As far as I can tell the only real solution here is one of the projects trying to rewrite core emacs so that it is better at concurrency, but who knows when or if any of those might bear fruit.
This said, I found the thing that irritated me the most about slowness in deft was the autosave happening too frequently. I increase the time on that to I think a minute and it helped. Also maybe cleanup your buffer list more frequently, that was a big one for me. I realize these are not actually solutions to what you're asking, but they might help.
Do I get this right? Looking at
deft.el, it looks like it's getting a list of files and then looks through these indeft-find-files.This could be sped up considerably if one uses an on-disk search like
ag/thesilversearcher. That's what @rene is doing in Sublimeless_ZK to great success. Not surprisingly, Emacs seems to sport compatible packages: https://cestlaz.github.io/posts/using-emacs-48-silversearcher/My Elisp is ... well, let's say I cannot write an
ag-based Deft that depends on file system data instead of an internal buffer cache. (That's what it seems to do, or else it wouldn't slow down Emacs completely, would it?)Author at Zettelkasten.de • https://christiantietze.de/
I think you are correct @ctietze . I actually do most of my searching with
helm-agrather than deft internal search.@pinkhurricanegown I don't have that many notes in deft, but could you elaborate on precisely when you experience slowness? When using its interface to filter out notes?
Could you describe if that still happens when you use
deft-filterto search rather than the usual deft interface?@EFLS I can speak to my own experience here, which is, yes, using the interface to filter notes is desperately slow (~900 notes for me). Initializing deft takes about 8 seconds. The first letter I type into the filter interface takes about 4 seconds to show up, and after that of course it speeds up dependent on how many results there are. Using
deft-filtersearch is perfectly reasonable.@EFLS
I think I'm about where @mediapathic is. I've got 880 zettels. Initializing deft doesn't take super long for me, for whatever reason (~2-3 seconds) -- but searching is just painful. It's not only the number of Zettels, though -- I'm like 80% sure it's down to the way the searching works in deft.
Every time you add or remove a character to your serach, the whole thing is locking briefly (or not so briefly) while it does the incrementing/decrementing. When I run
profilerand just do random searches, the report comes back with >45% of both the CPU and Memory time dedicated just todeft-filter-incrementwith something like 30% todeft-filter-decrement/deft-filter-decrement-word.By contrast, if you use
C-c C-l(by default) to calldeft-filter, you can do enter-your-search-hit-return-get-results searching, which removes all the slowdown -- or, at the very least, hides it behind an acceptable pause while it does its thing.From a user experience perspective, having a pause before displaying search results is vastly preferable to having input lag with every single keypress.
If I knew Elisp, I think the first project I'd take on is dismantling Deft to figure figure out how to add the option to have the search bar default to type-enter-results style searching, rather than incremental searching. Just adding that one option would fix most of my problems with Deft.
Either that, or I guess move to some other indexing back-end -- right now, Deft appears to just cat all the files in the directory into one giant string, and then searches that string for what you want.
I dunno!
The weird thing is, Notational Velocity/nvALT for example doesn't do anything more sophisticated than that: get the notes into memory and perform fast character-based search. Is it because there's no way to load stuff into Emacs memory without putting it into a buffer, and does that incur additional overhead? I don't know. But it's weird that this becomes an issue on modern day computing hardware even with an application core as old as Emacs's.
Author at Zettelkasten.de • https://christiantietze.de/
Totally! I used NVAlt for ages and ages, and never had this problem, even with a collection of notes twice what I've got now. It shouldn't be like this!
What makes it even more frustrating is, I know it's a problem with the Deft package and not Emacs itself.
Originally, all my zettels were individual headlines in one long Org file. I broke them out into individual files to move to Deft/Zetteldeft. When I re-combine all my current zettels back into an org file, it's only about 800kb -- we're not talking about a lot of data here, all told. Emacs can handle WAY bigger files like a champ.
When I dig around in that combined notes file using basic Emacs, there's no slowdown. Emacs only bites the dust when you're dealing with truly huge files, files with very long lines in them (1000s of characters long), or when you're applying a lot of custom font/color stuff with your theme. Or, when you're on Windows, there's an I/O bottleneck that limits the speed of certain kinds of processes.
And when I use other tools, there's no slowdown.
Take Swiper, for instance, which is a generic incremental search system that replaces the built-in search functionality across all of Emacs -- there's not the same lag when digging around in my zettels when they're one big file.
There's also
org-velocity, which comes with the full install of Org and replicates the Notational Velocity experience, incremental searching and all, without slowdown.So, why not use org-velocity or giant-file-with-Swiper? Because they're mediocre solutions. Deft/Zetteldeft is darn near perfect, except for this specific problem. And there aren't any better solutions for ZK on Windows -- much less solutions that will work across the Windows, macOS, and Linux machines I have to wander between for work.
Knowing what needs to be fixed, but not being able to fix it, is like having an itch somewhere that you can't reach -- maddening!
Have you tried Zettlr? It is cross-platform.
Maybe you could all drum up a Kickstarter campaign to pay the Deft author $1000 for better search
Author at Zettelkasten.de • https://christiantietze.de/
You're not wrong -- maybe it's worth seeing what kind of interest there would be in the emacs community.
Someone recently shared this interesting fork of
defton thezetteldefGithub repository: https://github.com/hasu/notdeftIt has the option of using a different backend for searching, which promises better speeds, as far as I understand it.
Although I don't plan -- at least not in the short term -- to integrate this into
zetteldeft, anyone can freely fork and tinkerWhat I really want here is
deftto have some kind of option to use whatever search strategy works for the user. So, default would be native deft search, but I could set a flag and useag, someone who likes xapian could use that, etc. I have no idea how difficult that would be to implement, so, no, I'm not volunteering to write itYeah, that does sound inviting -- the option, not the task to implement it (which would be way beyond of my coding skills).
Just a pointer for people stumbling upon this thread: discussion of Zetteldeft is available here https://forum.zettelkasten.de/discussion/1082/introducing-zetteldeft-an-emacs-package