Rich- yet plain-text notes using Marked and pandoc
Hi everyone,
for my Zettelkasten, I have some special requirements, namely:
- I need to include images
- I need to include references from a BibTex file
- I need to include formulas
However, storing notes in plain-text is crucial to me, for all the reasons that can be found on the blog and in the forum. I believe I would perceive and inline-image preview in a Markdown editor rather as a hindrance than as a feature.
The first requirement can be easily fulfilled by using the Markdown preview app Marked 2. Setting Marked as the external editor for The Archive, a view of the note with images is only one Cmd+Shift+E away.
The second and third requirement can be fulfilled by setting the (awesome) document converter pandoc as the custom processor in Marked.
In Marked's advanced settings, I enabled the custom processor, setting the path to "/usr/local/bin/pandoc" and the args to "-f markdown -t html5 -s --filter pandoc-citeproc --bibliography /path/to/bibliography.bib".
With this system, BibTex references (e.g. @bibtexkey) will the processed and formulas can be written using LaTeX syntax. I noticed some problems with inline formulas as in "we observe that $x_n$ is". Here, sometimes the indices are not set correctly. "Displayed formulas" surrounded by "$$" as seen in the attached image, everything seems to work perfectly.
I am quite happy with this setup, maybe it is helpful for some of you as well.
Howdy, Stranger!
Comments
@cdguit: Nice setup, thanks for sharing!
The only thing I would not be entirely happy with in the setup is that the bibliography data is not part of the stored zettel, but is just created when using the Markdown preview app. I would want the citation in the Bibliography section of the zettel. Trying to find out how to get that automated, possibly with a .bib-file as the source of my references (at the moment I'm using Zotero, but will probably switch soon).
I am doing pretty much precisely this. I don't have as much need for formulae and citations, but for inline images I work in parallel in either The Archive or Emacs with Marked open next to it. Especially makes doing things like making sure tables and other formatting looks good for client worke.
@Vinho
I don't know how to do what you want, but there should be a solution that is not to complicated. I was looking for a CLI tool to query .bib files, but didn't find anything sophisticated.
@mediapathic
Yes, tables and syntax highlighting look also very nice with Marked! I really like this separation of plain-text content and a pretty html view.
@cdguit: Yesterday I had a closer look at your described setup and I must say that I like a lot about it and decided to go for it (or something similar) as well. I decided that it's not necessary to have the bibliography data in the original plain text file – the citekey and the associated .bib-file make it resilient enough. Bibliography data is mainly important if I want to share zettels with others that don't have the .bib-file, so if I create a nice file to share with Marked 2 and it creates the bibliography for me automatically that's perfect.
Until now I used Zotero to manage my references and I really appreciate the gain in customisability and control when it comes to entry types and their fields that goes with switching to BibDesk, although I have to spend some time getting into all of that. Ideally, I would like BibDesk to just show the entry types and fields that I really use and consider necessary. This would mean deleting or changing a lot of the default BibTeX types and adding new ones (like "presentation" or "podcast"). This not only seems to create a few problems with BibDesk (a lot of crashes during the process), but also later in your workflow, because Pandoc doesn't recognise all the new custom fields I'm inventing (e.g. "Speaker" in the entry type "presentation"). Do you (or does anyone else) know where I would need to change what in order to make that work? And is there a possibility to change the citation style for the bibliography entries somewhere? Ideally, I would be able to determine exactly how I want it cited for each of my custom entry types...
@Vinho, I wouldn't recommend to invent your own entry types and fields, esp. not with BibTeX whose format has been fragmented enough already (due to the lack of a proper spec and coordinated development of the format).
Instead, it may be worth looking at richer bibliographic exchange formats such as BibLaTeX or CSL/Citeproc JSON (or its YAML variant). These formats should be also supported by pandoc.
https://pandoc.org/MANUAL.html#citations
You should be able to modify the citation style via the
--csloption:https://pandoc.org/MANUAL.html#citation-rendering
For more info on CSL, see:
https://citationstyles.org
@msteffens: Thanks for your informative post!
Am I right in saying that you advise against inventing own types and fields and instead would recommend using a format that already provides suitable types and fields out of the box?
BibLaTeX or CSL JSON offer more types, but neither of them offer all I want... In general, I'd much prefer it if I could freely invent my own types and fields and didn't have to try to fit my stuff into a system that wasn't really made for it. Do you know more specifically which problems arise when I invent my own types and fields and if these problems can be avoided somehow?
Yes, exactly.
Yes, I can understand that. Still, I'd question myself if defining your own custom types & fields is really worth the hassle. What do you gain with this, and where will you exactly benefit by using your own types/fields. Could you maybe gain almost the same benefits by reusing general fields in a creative way? E.g., could you instead add special keywords to the notes or tags fields?
You loose any kind of interoperability – this not only regards interoperability with other users but also with your future self and with using different tools in the future. With the standard bibliographic exchange formats, interoperability is a tricky thing and often quite hard to achieve. The bibliographic formats aren't standardized enough, and everybody likes to use or interpret them slightly differently. The world of bibliographic exchange formats is a huge mess already, and a PITA for everyone trying to improve interoperability (I've been working in the field for >15 years so my opinion is probably biased).
Sure, if you will only ever use this for yourself, then, of course, you're free to do whatever you want. But, as you noted, you also want to use tools like Zotero, BibDesk or Pandoc etc. And any bigger customization makes it harder to use such tools for your benefit.
@cdguit:
Thank you again for posting this and your helpful advice. I've decided to go with the same system as you – so am now working with Pandoc and Marked 2. Also created my own csl-style, so the citations look perfect (except from a few small problems that I still need to solve).
Overall, it seems to me like a very elegant and easy-to-use solution – I'm very pleased with it.
@msteffens:
Thank you for your important advice as well, I've decided to use BibLaTeX with BibDesk for now and I'm trying to use a small amount of predefined entry types for everything I have – working with the field "type" to distinguish between different kinds of resources. If it was easier to find out how exactly pandoc-citeproc parses the BibLaTeX entry types and fields, I might refine this a bit more, but for now that'll do.
@Vinho:
Alright, I am glad the system works for you!
When you're satisfied with your csl-style, feel free to share it! Up to now, I didn't really care about the look, but when you have something that looks nice, I'm curious to try it out :-)
@msteffens:
Thank you for your contributions! I should have been more precise, when writing in LaTeX I am actually using BibLaTeX, I just (incorrectly) keep calling it BibTeX.