couteausuis.se thoughts

bpg dump

05 Feb 2019

In the past few weeks, i stumbled upon a vast trove of images that needed to be ordered and filled. Here’s the problem : To save bandwidth, a genius converted all those images into the BPG format.

bpg problem(s)

First off, the format & thinking behind the BPG project is fantastic. Furthermore, Fabrice Bellard is the perfect man to work on a Library like that. All around the web, you can find promising results : savings of up to 80% while retaining the same quality as a JPEG at Q80.

Now, the first problem with a new (better) format is that from now on, it adds another player to the Monopoly game (relevant XKCD). Second, even after more than 9 months has passed since its release, the bpg format is not supported by any “big” image viewer. To give you an idea, here’s the list of apps that supports it :

solution one

I reluctantly decided to convert the images back to jpg. There are more than 45k files… In my search for a (free) converter that would support directly BPG -> JPG, I found Romeolight BPGconv. Here’s the tricky part : I need a windows VM to run it, with enough space to hold the (non/)converted files. I set up a shared network folder to hold the bpg files (30gb) and another one with about 80gb free to hold the jpg files. I then loaded all the files into the batch decoder and left it ran the whole night.

The RomeoLight BPGconv suffers from a big bug where it copies every bpg file worked on into the AppData/Temp folder but doesn’t remove it afterwards. This bug leads to a full disk and the whole VM stopping after only 1h.

To circumvent that, I wrote a simple BAT script that deletes the Temp folder, and use Task Scheduler to run it every 30 minutes. It worked and was surprisingly stable for 12hrs. After 12hrs with the frankenstein computer heating my room, I extrapolated the numbers and calculated that I needed 4.5 more days for it to churn through all the images. That’s a big no.

solution two

Luckily for me, libbpg is available on Homebrew and it comes with a bpg decoder bpgdec. You can use it to convert BPG files to PNG (very long process) or to PPM, which seems faster. What the heck if PPM ? At this point I feel like I’ve fallen down a rabbit hole. PPM stands for Portable Pixmap iMage and is really 24-bit image data in a text file… very helpful. From a 350kb BPG, we get a 12mb PPM file.

To convert my fresh PPM file into something useable (that was the goal of the post) I need another userful tool, imagemagick.

I wrote a small bash script to loop through all the bpg files in a folder and convert them to jpg.

#!/bin/bash
dir=$(pwd)
for fullfilename in $dir/*.bpg; do
    [ -e "$fullfilename" ] || continue
    filename=$(basename -- "$fullfilename")
    filename="${filename%.*}"
    echo "Converting image # ${filename}"
    bpgdec -o $filename.ppm $fullfilename
    convert -quality 75 $filename.ppm out/$filename.jpg
    rm $filename.ppm
done

final thoughts

This bash script is not very parallell-friendly (are any bash scripts?), so I split my stack of 45k images into 8 folder and started 8 scripts separately (4 cores + hyperthreading). In 8hrs everything was done, and I was very happy. I guess the moral of this post is to always try to do it yourself, you’ll learn much more along the way and probably find shortcuts.

Related Posts