Possible Interest: A new scientific and technical publishing system built on Pandoc

jamesrregan

I recently came across a markdown based publishing system that may be of interest to people on this forum. I have not tried it, but it looks interesting and worthy enough to share. The architecture is language agnostic.

Quarto - scientific and technical publishing system built on Pandoc

https://quarto.org/

Christoph

Oh wow, I hadn‘t heard of that before, thanks. In some aspects quite close to orgmode but certainly different enough to be worth a look.

msteffens

Quarto looks indeed very impressive and their Projects feature (which can also generate entire websites or books, see examples) screams for integration with a knowledge management system.

ZettelDistraction

Thanks for this. It turns out that I need to start using Jupyter notebooks. I'll check whether Quarto can be integrated with Zettlr as an export format.

Jvet

I just started using it, I have to say I really like it. Specially because I can embed R code and many other things.

ZettelDistraction

I have not been successful integrating Quarto with Zettlr so far. I have been busy customizing the Pandoc latex template that comes with Zettlr. However, my efforts with LaTeX and Zettlr have been hampered as the documents become larger than a screenful (a Zettel). Zettlr's built-in LaTeX interpretation slows editing down to a crawl as the combination Markdown+LaTeX documents become longer and more complicated. These are documents that wouldn't render in KaTeX or MathJax anyway. I might as well stick with WinEDT. This is what I get for writing a longer-form document instead of a Zettel in Zettlr. Projects have to be done outside the Zettelkasten editor. I will revisit Quarto, use another editor, or ask the Zettlr developer if I could turn off LaTeX interpretation and leave that to the export process... WYSIWYG still doesn't work well with document processing.

This is a case of being pulled in different directions by software. The final product should be a LaTeX source file (or several) together with a PDF document. The Zettelkasten has its editor, Zettlr, in which I started a project together with a Jupyter notebook running under SageMath for some of the exploratory calculations that couldn't be done by hand. Jupyter in turn can be integrated with Quarto, which might give acceptable LaTeX, or else I could use SageMath LaTeX, which I haven't tried because I'm shy .

As the project grows, so does the need for LaTeX customization of Zettlr (and to a lesser extent the Pandoc LaTeX template, not to mention MikTeX itself for my local customizations of LaTeX). I check my LaTeX customizations of Zettlr and Pandoc with the Markdown to LaTeX source Pandoc filter, which either fails if there is a YAML/Pandoc error or else brings up WinEDT with the converted source file, which still may have errors. WinEDT is indispensable for indicating where my LaTeX macros and code have gone wrong. However, the TeXify command in WinEDT can produce substandard PDF output (inferior page layout -- too much space). Better output is available from Zettlr's Pandoc to PDF filter, which will produce superior PDF output (no extra space in the page output) and will bring up the Adobe PDF reader. Consequently, the LaTeX/Pandoc customizations for LaTeX source have to be maintained in parallel with the customizations for PDF output. This won't render in KaTeX or MathJax (not without a tremendous detour that may go nowhere), so I have given up with the HTML Pandoc filter.

The point is that the workflow for this hasn't settled. Some of the results from the Jupyter calculations should be fed back into the Zettelkasten and into the paper. And because it can be easy to forget what the customizations were and why they were done the way they were done, the customizations and the workflow they enable should be fed back into the Zettelkasten. Some of this benefited from @ctietze 's critique of one of my macro Zetteln--the link is to the revision. In addition to the technical document processing, the project requires reading pages of theoretical mathematics and interpreting this in terms of the project and conversely, and some of that also goes into the Zettelkasten. The quantity of mathematics needed required a new approach--one I probably should have learned in graduate school, but my policy now is to blame most everything prior CPAP therapy on obstructive sleep apnea and go forward.

This practice of manipulating one's personal narrative is known as "story editing." Personal narratives are false in any case, and some of us don't even have a coherent personal narrative. Story editing has its advantages, nevertheless. Anyway, the new approach doesn't work without a Zettelkasten, for me anyway.

And that means I need a new cartoon. This is the more complete answer to @Will 's what are your plans for the coming week post, incidentally.

Loni

If I understand your problem in the right way, you may have :

• Something to take notes in LaTeX for maths notes, and for LaTeX notes to remember;
• Something to edit and publish final documents from LaTex to Pandoc.

Is it so?

Maybe you should consider redacting your final documents into VisualStudio Code or Sublime Text, or building your whole Zettelkasten with VSC, Foam, Pandoc and LaTeX plugin ? VSC also have a Quarto plugin as well and have some plugins for Zotero integration too.

I think I would choose this solution if I was in your position (and I would be there too if I didn't need medias insertions). Zettlr is a nice editor, but I find it quite stiff to handle. The way it handles screen display of long text seems delayed, I don't like it. You would benefit of the advantages of using and IDE like the "Go To" shortcuts, fast doc opening, command palette at your fingers. Foam is fine for a Zettelkasten too, seems to do the jo even if our needs might differs, it offers wikilinks and tags display.

I hope it helps !

ZettelDistraction

@Loni said:
If I understand your problem in the right way, you may have :

• Something to take notes in LaTeX for maths notes, and for LaTeX notes to remember;
• Something to edit and publish final documents from LaTex to Pandoc.

Is it so?

>

Most often

paper --> Markdown+LaTeX (in Zettlr) --> Pandoc --> LaTeX --> WinEDT --> PDF
or
paper --> Markdown+LaTeX (in Zettlr) --> Pandoc --> LaTeX --> PDF
or
nothing or paper or ZK --> SageMath + Jupyter notebook --> Jupyter notebook --> nowhere so far
or
paper --> nowhere (latex can be time-consuming. It helps to have a library of forms, but even then it can be slow going)
or
nothing --> nowhere (this is where the bulk of "activity" occurs).

I have Scrivener 3, which is beginning to develop bit rot from lack of use.

Maybe you should consider redacting your final documents into VisualStudio Code or Sublime Text, or building your whole Zettelkasten with VSC, Foam, Pandoc and LaTeX plugin ? VSC also have a Quarto plugin as well and have some plugins for Zotero integration too.

I'll look into this. My ZK uses at least two features of Zettlr:
1. The ID generation [Foam uses wikilinks and does away with IDs]
2. The backlink capability [I see that Foam has this]

I think I would choose this solution if I was in your position (and I would be there too if I didn't need medias insertions). Zettlr is a nice editor, but I find it quite stiff to handle. The way it handles screen display of long text seems delayed, I don't like it.

Even without mathematics--then I'm screwed. Could be because it's an electron app...

You would benefit of the advantages of using and IDE like the "Go To" shortcuts, fast doc opening, command palette at your fingers. Foam is fine for a Zettelkasten too, seems to do the jo even if our needs might differs, it offers wikilinks and tags display.

I hope it helps !

Very helpful!

Loni

I'll look into this. My ZK uses at least two features of Zettlr:
1. The ID generation
2. The backlink capability

Foam handles very well the backlings for sure, it has a backling sidebar tool to follow them as well. I made tests this morning : it handles from different folders, recognize the wikilinks format too.

I think you can use a Timestamp ID with a format you'll choose like "YYMMddHHmmSS" with this tool , or this one seems fine too.

I use Obsidian Timestamp plugin to generate ID too when creating files from a structure note. You can also create a new file, insert the timestamp in it and create the filename with a quick CntrC, CntrV.

Even without mathematics--then I'm screwed. Could be because it's an electron app...

Naay sire, nay !
I've tested Obsidian, electron as well, in an awfull stretch. 80.000 words was the limit on my previous computer, an old machine with an old AMD cpu and 8go RAM. Now, I have a note with a dozen heavy and biiiig pictures and Live Preview, works fine.
Visual Studio is an electron app itself and it goes smoothly with really big documents as well.
Typora is electron too and people seems happy with big text files too.

I am not a developper myself, but I think there is a problem in the way Zettlr handles large files. However, it becomes better as time pass. LaTex is not the average user need and does not benefit from the tests of a large users set.

ZettelDistraction

Our comments crossed -- I updated mine after I looked at Foam.

Very helpful!

Foam looks good actually. I'll try VS Code with plugins in parallel and see what it costs in time and effort to move over. My customizations will probably transfer over, since they are input files to Pandoc. Depending on the Pandoc filter, I go to LaTeX source or to PDF directly.

Loni

Very helpful!

Happy Loni !
I just finish the tests of the first timestamp plugin, it works well !

Foam looks good actually. I'll try VS Code with plugins in parallel and see what it costs in time and effort to move over. My customizations will probably transfer over, since they are input files to Pandoc. Depending on the Pandoc filter, I go to LaTeX source or to PDF directly.

Pandoc uses its own set of files for customizing output. Zettlr helps with a human usable interface, but I know you can create your own set of instructions and communicate with Pandoc thanks to command lines. Zettlr may have created those files as well, or you would find them in a "Zettlr" subfolder (C://User/Application/Roam/Zettlr for example). You can copy and add them to Pandoc.

I didn't test Pandoc VSC plugins, as I don't have the same needs, they wouldn't be accurate for your uses. But they exists and may help you.

ctietze

@ZettelDistraction Ah well this might get me into hell eventually, but maybe you hate yourself enough to pick up Emacs and use org-roam for your Zettelkasten and abuse the tight integration with, well, anythin in Emacs, including LaTeX? There's ways to make formulaic input and preview nicer, too: https://karthinks.com/software/latex-input-for-impatient-scholars/

*ducks*

Loni

@ctietze said:
@ZettelDistraction Ah well this might get me into hell eventually, but maybe you hate yourself enough to pick up Emacs and use org-roam for your Zettelkasten and abuse the tight integration with, well, anythin in Emacs, including LaTeX? There's ways to make formulaic input and preview nicer, too: https://karthinks.com/software/latex-input-for-impatient-scholars/

*ducks*

If only I can understand what I do in Emacs, I would pick this solution myself ! T_T Orgmode is incredibly powerful, and Emacs is a kind of OS by itself.

ZettelDistraction

@ctietze said:
@ZettelDistraction Ah well this might get me into hell eventually, but maybe you hate yourself enough

Easily more than enough. As for Hell, if you're going then I'm definitely going.

to pick up Emacs and use org-roam for your Zettelkasten

Possible. I've flirted with Emacs on occasion.