Colored clipboard from shell

In August 2018 I was evaluating Ceph file system for my servers and writing a blog post about it. So I copied the shell commands and the console output to WordPress. To my astonishment it is quite difficult to copy paste from a terminal to another document or in this case my blog without losing formatting.
Believe me I tried it. If I just copy paste from my terminal into the edit blog view I get dull black text.

How this is usually handled with WordPress

You can insert code in
<code></code> tags for an output like this:

 class lol : test { public: inline void main (char[]* args){ } }; 

or you can use plugins like this one based on SyntaxHighlighter¹⁾http://alexgorbatchev.com/SyntaxHighlighter/, that parse and format copied source code to style pasted code:

[cpp]
class lol : test {
public:
inline void main (char[]* args){ }
};
[/cpp]

becomes

class lol : test {
public:
inline void main (char[]* args){ }
};

And

[bash] ~/ ls -la
 total 664
 drwxr-xr-x 44 markus markus  12288 Jun  5 23:48 .
 drwxr-xr-x  3 root   root     4096 Mar  3  2017 ..
 drwxrwxr-x  6 markus markus   4096 Apr 14 07:25 ansifilter-src
 -rw-rw-r--  1 markus markus    653 May  4  2018 .anyconnect
 -rw-------  1 markus markus   7092 Mar  9  2017 .bash_history[/bash]

becomes

  ~/ ls -la
 total 664
 drwxr-xr-x 44 markus markus  12288 Jun  5 23:48 .
 drwxr-xr-x  3 root   root     4096 Mar  3  2017 ..
 drwxrwxr-x  6 markus markus   4096 Apr 14 07:25 ansifilter-src
 -rw-rw-r--  1 markus markus    653 May  4  2018 .anyconnect
 -rw-------  1 markus markus   7092 Mar  9  2017 .bash_history 

There exist many similar tools²⁾https://stackoverflow.com/questions/1647724/color-syntax-highlighting-within-an-html-code-tag , https://github.com/google/code-prettify, https://askubuntu.com/a/405970/299013. Another popular answer seems to be to copy it into a text editor and then recolor the code³⁾https://stackoverflow.com/a/51043085/643011 ,
https://askubuntu.com/a/394230/299013.

The way to get this into WordPress is to export it as HTML formatted text and then include the HTML-colored text into WordPress. This works well.

But that's odd. Why would I throw away the color from the terminal just to reconstruct the color with a plugin? I want to include the original color in my blogpost.

Digging deeper

Unix/Linux based text terminals use ANSI escape sequences to show colored text. ⁴⁾https://en.wikipedia.org/wiki/ANSI_escape_code#Colors ⁵⁾http://www.lihaoyi.com/post/BuildyourownCommandLinewithANSIescapecodes.html#colors

ANSI escape sequences are started by the ASCII escape character hex
\x1B ( in decimal: 27 visualized as ^ ) followed
by [ and multiple bytes encoding the actual ANSI control sequence.

Whats the difference between ANSI control sequence and ASCII code? While there is an ASCII code for making a beeping sound, to signal the end of a column or to shift the cursor to the next line, there are only so many commands that can fit into a few characters in 7bit ASCII.⁶⁾https://www.cs.cmu.edu/~pattis/15-1XX/common/handouts/ascii.html Multi-Byte ANSI control sequences are more powerful and can clear the screen, change the font and font style, save or change cursor position, and use up to 24bit background and foreground text color in the virtual terminal.

Your favorite programming language probably has convenience libraries for printing colored text⁷⁾https://stackoverflow.com/questions/287871/how-to-print-colored-text-in-terminal-in-python,
https://stackoverflow.com/questions/5947742/how-to-change-the-output-color-of-echo-in-linuxMost GNU core utils have colored support⁸⁾https://askubuntu.com/questions/17299/what-do-the-different-colors-mean-in-ls However how do I get this into my browser?

Finding the culprit

I tried all combinations of trying to copy colored output with

1. the X-based GNOME Terminal and
2. XTerm
3. terminator
4. guake

pasting into a text editor

1. Libreoffice Writer
2. MS Word and
3. gedit

or into a browser where I tried both

1. Firefox and
2. Chromium

while using as a target

1. WordPress Gutenberg visual editor or
2. https://www.tiny.cloud/ rich HTML textfield or
3. https://quilljs.com/ rich HTML textfield

All while trying both the

1. PRIMARY X-selection (ICCCM) way of copying (middle mouse button)
2. and CLIPBOARD way of copying.

(Yes, X.org is weird.⁹⁾ https://specifications.freedesktop.org/clipboards-spec/clipboards-latest.txt
https://wiki.archlinux.org/index.php/Clipboard#
https://www.jwz.org/doc/x-cut-and-paste.html
https://unix.stackexchange.com/a/254745/34172 )

All failed.

So is it the terminals, X or the target applications that fail?

You can redirect the colored output to a file ls -la --color=always > test.txtand later view it colored just fine with cat test.txt in the terminal. If it doesn't work with your cat, you might have to use echo -e $(cat test.txt) instead¹⁰⁾https://unix.stackexchange.com/a/388747/34172

Opening the test.txt file in Open Office Writer yields color stripped text. While chrome doesn't know how to render ASCII escape character and shows garbled output. But this is a better result than before for the browser. At least by piping to a file we don't get the color code stripped completely.

aha - to convert to html

We can successfully get colored text out from the terminal by piping data through aha ("Ansi HTML Adapter"). aha creates an HTML file.
ls -la --color=always | aha > ls-output.htm¹¹⁾https://stackoverflow.com/questions/245121/a-library-to-convert-ansi-escapes-terminal-formatting-color-codes-to-html

The output is not amazing HTML, every line is encapsulated in a span with a simple style attribute, but it works!

Copy to clipboard with xclip use file MIME type

There is an alternative way to copy to clipboard using the Xclip utility. ¹²⁾https://unix.stackexchange.com/a/145134/34172
https://stackoverflow.com/questions/4486376/does-the-gnome-clipboard-have-a-mime-type-associated-with-the-data
https://stackoverflow.com/questions/749544/pipe-to-from-the-clipboard-in-bash-script
The ICCCM standard allows to supply multiple types of file representations, ¹³⁾https://tronche.com/gui/x/icccm/sec-2.html#s-2.6.2 each identified by its MIME File type. This allows you to e.g. copy and paste picture binary data. If I remember correctly from the dark days where I programmed Swing desktop applications in Java, there is a similar mechanism in all OSes and that concept was also exposed in the Java cross-platform copy paste API.

We can now use this and generate a format that can successfully be pasted into Open Office Writer:

ls --color=always | aha | xclip -selection clipboard -t 'text/html' -i

Alternative using RTF

An alternative way is providing the pasted into application an RTF formatted, colored text. For this I downloaded and built ansifilter, which can do the RTF conversion: https://github.com/andre-simon/ansifilter
http://www.andre-simon.de/doku/ansifilter/en/ansifilter.php

With this you can ls -la --color=always | ansifilter --rtf | xclip -selection clipboard -t 'text/rtf' -i

This again is accepted by Open Office Writer, but not any browser and neither gedit.

Fun fact, most of the tools you find are for stripping ANSI codes and removing RTF files. Because progress bar cursor annotations and other stuff is annoying in logs

There is a tool for the Windows Powershell to copy colorized text to RTF formatted output, but I haven't tried it.¹⁴⁾https://devblogs.microsoft.com/powershell/colorized-capture-of-console-screen-in-html-and-rtf/
https://superuser.com/questions/77781/how-to-copy-text-from-the-cmd-console-and-keep-the-formatting-with-color

Tying it all together

In order to copy from the shell you need to redirect all output from the shell:
zsh |& tee zsh.log
aha -f zsh.log > zsh-output.html

Or from ssh (same concept as above):
ssh [email protected] | tee ssh3.log
This works great!

You can also directly copy to the clipboard and paste into the browser or an editor with xsel: ls --color=always | aha | xsel -i -b ¹⁵⁾https://askubuntu.com/a/394612/299013

WordPress code editor just inserts the HTML into the document as expected. If on the other hand you paste the HTML source into the visual editor it converts the pasted HTML to styled text (keeping the font bold) but strips the color. 🤦🏾‍♂️

Bonus: GNU Screen

But what if I already have some output in the screen copy buffer that I would like to copy?

you can use hardcopy -h <filename>.
¹⁶⁾https://stackoverflow.com/a/6604296/643011
But this strips ANSI color codes before saving to file. No color 🙁

You can select a range from the buffer to be copied.
Then set a buffer file with bufferfile /tmp/somefile.txt
¹⁷⁾https://stackoverflow.com/a/5896568/643011
And finally use the command >
But this also writes to the file without color.

You can also use :writebuf /tmp/somefile.log - same result: colors are not preserved!

Oddly enough screen's log on does preserve colors! But you have to enable it before the output occurs!
This file is saved to screens working directory. Even though you can't easily put it into screen.rc it is possible to enable it by default.

The disadvantage of this method is that this produces 100s MB big log files and HTML files. And some text editors and browsers choke several minutes opening that. VS Code works great by the way.

Bonus: MacOS Terminal

Apparently some os x terminals can copy paste out of the box¹⁸⁾https://superuser.com/a/879099/182880

And it does work with:
Textedit Version 1.13 (333)
Terminal Version 2.8.3 (404.1)
But not with the Notes app

Markuss-Mac:~ markuskrainz$ ls -G
Desktop Documents Downloads Library Movies Music Pictures Public

Summary

The problems seem to stem from the fact that terminal color is an ANSI escape sequence, that is not understood by most programs, and because it also contains potentially confusing characters such as moving the cursor, it is usually stripped before copying. A better way would be to leave the colors alone / and or offer multiple formats to copy from (see how the macOS terminal does this)

I can redirect (tee) all my ssh connections output to a file, convert this to HTML-formatted color. Then I have to manually scroll through my copy buffer to get all the history. Finally I can insert the code into WordPress blog using the code editor.

If you have a suggestion on how to copy directly from the screen copy buffer without losing formatting then please send me a message!

Final Result

root@f-serv:/mnt# vim /etc/fstab
"/etc/fstab" 6L, 382C
# UNCONFIGURED FSTAB FOR BASE SYSTEM
UUID=e78534f3-ff39-4b37-9dc0-89c6ebe27951 /boot ext2 defaults 0 2
PARTUUID=de00fcfe-5c5e-4a72-8e4d-e23a149610ee /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1
PARTUUID=bc16cfe9-6286-40fd-bba2-ac44620c7018 /boot/efi2 vfat nofail,x-systemd.device-timeout=1 0 1
rpool/var/log /var/log zfs defaults 0 0
rpool/var/tmp /var/tmp zfs defaults 0 0