reference management startup time?

zk_1000

I'm wondering about scalability of reference manager. How big can they grow before getting slow? I'm using Zotero with a small collection, but add ons making startup time a bit slow already.

What is your experience, how much entries do you have and how much do you think your setup can handle?

Andy

I use BibDesk (plus Zotero, which I use only as a web scraper via Zot2Bib—the publications are not stored in Zotero) with tens of thousands of publications. Startup time is long enough that I only start the app once per day and keep it open all day. It's very fast after startup because all publications are loaded into RAM. (Make sure that you have plenty of RAM!)

As an experiment, I once imported my BibDesk database into Zotero + Better BibTeX to compare speed, and Zotero was slower but still usable. If I used Zotero, I would do the same that I do with BibDesk: open it at the start of the day and keep it open all day if I planned to use it. I read somewhere that Zotero's data model will be changing, so it may be faster in future versions.

I wouldn't worry about scalability unless you are planning to work with hundreds of thousands or millions of publications.

ctietze

We have a repository of 10k Markdown files for testing apps, but not for 10k BibDesk entries

If anyone has the time to generate that many, I'd be curious to benchmark performance!

Andy

@ctietze: Nelson Beebe has an extensive collection of BibTeX bibliographies on his website. Just a few of those will easily add up to 10k entries or more.

EDIT: And Brad Weslake's philosophy bibliography is a BibTeX file, created with BibDesk, that is larger than 10k entries. We previously discussed that the git repository for that file is enormous, probably due to BibDesk's hex-encoded bdsk-file-n fields.

digbyphonic

@zk_1000

i have used ZOTERO daily since 2020. now at 5440 entries. sync'd installs on 5 PC's. i would not want to be without it. current setup uses a plugin (BetterBibTeX) to automatically update a JSON file. this is read by a Markdown editor of choice. all stable, as far as i can tell.

i hope this helps, and is an answer you were looking for.
all the best/

runit

I use Zotero for years now. I have over 1000 items, a lot of them with pdfs, snapshots or other attachments and I have not noticed a slow down yet.

Actually since I am using the beta of Zotero 7, the speed and overall "smoothness" has (imho) increased significantly. So I am also confident, that it will hold up in the future.