369 lines
13 KiB
Markdown
369 lines
13 KiB
Markdown
---
|
|
title: Obscure features of JPEG
|
|
author: Chris Hodapp
|
|
date: "2011-11-24"
|
|
tags:
|
|
- Technobabble
|
|
- jpeg
|
|
- image_compression
|
|
---
|
|
|
|
*(This is a modified version of what I wrote up at work when I saw
|
|
that progressive JPEGs could be nearly a drop-in replacement that
|
|
offered some additional functionality and ran some tests on this.)*
|
|
|
|
Introduction
|
|
============
|
|
|
|
The long-established JPEG standard contains a considerable number of
|
|
features that are seldom-used and sometimes virtually unknown. This
|
|
all is in spite of the widespread use of JPEG and the fact that every
|
|
JPEG decoder I tested was compatible with all of the features I will
|
|
discuss, probably because [IJG libjpeg](http://www.ijg.org/) (or
|
|
[this](http://www.freedesktop.org/wiki/Software/libjpeg)) runs
|
|
basically everywhere.
|
|
|
|
Progressive JPEG
|
|
================
|
|
|
|
One of the better-known features, though still obscure, is that of
|
|
progressive JPEGs. Progressive JPEGs contain the data in a different
|
|
order than more standard (sequential) JPEGs, enabling the JPEG decoder
|
|
to produce a full-sized image from just the beginning portion of a
|
|
file (at a reduced detail level) and then refine those details as more
|
|
of the file is available.
|
|
|
|
This was originally made for web usage over slow connections. While it
|
|
is rarely-used, most modern browsers support this incremental display
|
|
and refinement of the image, and even those applications that do not
|
|
attempt this support still are able to read the full image.
|
|
|
|
Interestingly, since the only real difference between a progressive
|
|
JPEG and a sequential one is that the coefficients come in a different
|
|
order, the conversion between progressive and sequential is
|
|
lossless. Various lossless compression steps are applied to these
|
|
coefficients and as this reordering may permit a more efficient
|
|
encoding, a progressive JPEG often is smaller than a sequential JPEG
|
|
expressing an identical image.
|
|
|
|
One command I've used pretty frequently before posting a large photo online is:
|
|
|
|
{{<highlight bash>}}
|
|
jpegtran -optimize -progressive -copy all input.jpg > output.jpg
|
|
{{< / highlight >}}
|
|
|
|
This losslessly converts *input.jpg* to a progressive version and
|
|
optimizes it as well. (*jpegtran* can do some other things losslessly
|
|
as well - flipping, cropping, rotating, transposing, converting to
|
|
greyscale.)
|
|
|
|
Multi-scan JPEG
|
|
===============
|
|
|
|
More obscure still is that progressive JPEG is a particular case of
|
|
something more general: a **multi-scan JPEG**.
|
|
|
|
Standard JPEGs are single-scan sequential: All of the data is stored
|
|
top-to-bottom, with all of the color components and coefficients
|
|
together and in full. This includes, per **MCU** (minimum coded unit,
|
|
an 8x8 pixel square or some small multiple of it), 64 coefficients
|
|
each for each one of the 3 color components (typically Y,Cb,Cr). The
|
|
coefficients are from an 8x8 DCT transform matrix, but they are stored
|
|
in a zigzag order that preserves locality with regard to spatial
|
|
frequency as this permits more efficient encoding. The first
|
|
coefficient (0) is referred to as the DC coefficient; the others
|
|
(1-63) are AC coefficients.
|
|
|
|
Multi-scan JPEG permits this information to be packed in a fairly
|
|
arbitrary way (though with some restrictions). While information is
|
|
still stored top-to-bottom, it permits for only some of the data in
|
|
each MCU to be given, with the intention being that later scans will
|
|
provide other parts of this data (hence the name multi-scan). More
|
|
specifically:
|
|
|
|
* The three color components (Y for grayscale, and Cb/Cr for color) may be split up between scans.
|
|
* The 64 coefficients in each component may be split up. *(Two
|
|
restrictions apply here for any given scan: the DC coefficient must
|
|
always precede the AC coefficients, and if only AC coefficients are
|
|
sent, then they may only be for one single color component.)*
|
|
* Some bits of the coefficients may be split up. *(This, too, is
|
|
subject to a restriction, not to a given scan but to the entire
|
|
image: You must specify some of the DC bits. AC bits are all
|
|
optional. Information on how many bits are actually used here is
|
|
almost nonexistent.)*
|
|
|
|
In other words:
|
|
|
|
* You may leave color information out to be added later.
|
|
* You may let spatial detail be only a low-frequency approximation to
|
|
be refined later with higher-frequency coefficients. (As far as I
|
|
can tell, you cannot consistently reduce grayscale detail beyond the
|
|
8x8 pixel MCU while still recovering that detail in later scans.)
|
|
* You may leave grayscale and color values at a lower precision
|
|
(i.e. coarsely quantized) to have more precision added later.
|
|
* You may do all of the above in almost any order and almost any
|
|
number of steps.
|
|
|
|
Your libjpeg distribution probably contains something called
|
|
**wizard.txt** someplace (say, `/usr/share/docs/libjpeg8a` or
|
|
`/usr/share/doc/libjpeg-progs`); I don't know if an online copy is
|
|
readily available, but mine is
|
|
[here](<./libjpeg-wizard.txt>). I'll
|
|
leave detailed explanation of a scan script to the "Multiple Scan /
|
|
Progression Control" section of this document, but note that:
|
|
|
|
* Each non-commented line corresponds to one scan.
|
|
* The first section, prior to the colon, specifies which plane to
|
|
send, Y (0), Cb (1), or Cr (2).
|
|
* The two fields immediately after the colon give the first and last
|
|
indices of coefficients from that plane that should be in the
|
|
scan. Those indices are from 0 to 63 in zigzag order; 0 = DC, 1-63 =
|
|
AC in increasing frequency.
|
|
* The two fields immediately after those specify which bits of those
|
|
coefficients this scan contains.
|
|
|
|
According to that document, the standard script for a progressive JPEG is this:
|
|
|
|
{{<highlight text>}}
|
|
# Initial DC scan for Y,Cb,Cr (lowest bit not sent)
|
|
0,1,2: 0-0, 0, 1 ;
|
|
# First AC scan: send first 5 Y AC coefficients, minus 2 lowest bits:
|
|
0: 1-5, 0, 2 ;
|
|
# Send all Cr,Cb AC coefficients, minus lowest bit:
|
|
# (chroma data is usually too small to be worth subdividing further;
|
|
# but note we send Cr first since eye is least sensitive to Cb)
|
|
2: 1-63, 0, 1 ;
|
|
1: 1-63, 0, 1 ;
|
|
# Send remaining Y AC coefficients, minus 2 lowest bits:
|
|
0: 6-63, 0, 2 ;
|
|
# Send next-to-lowest bit of all Y AC coefficients:
|
|
0: 1-63, 2, 1 ;
|
|
# At this point we've sent all but the lowest bit of all coefficients.
|
|
# Send lowest bit of DC coefficients
|
|
0,1,2: 0-0, 1, 0 ;
|
|
# Send lowest bit of AC coefficients
|
|
2: 1-63, 1, 0 ;
|
|
1: 1-63, 1, 0 ;
|
|
# Y AC lowest bit scan is last; it's usually the largest scan
|
|
0: 1-63, 1, 0 ;</pre>
|
|
{{< / highlight >}}
|
|
|
|
|
|
And for standard, sequential JPEG it is:
|
|
|
|
{{<highlight text>}}
|
|
0 1 2: 0 63 0 0;
|
|
{{< / highlight >}}
|
|
|
|
|
|
In
|
|
[this image](./20100713-0107-interleave.jpg)
|
|
I used a custom scan script that sent all of the Y data, then all Cb,
|
|
then all Cr. Its custom scan script was just this:
|
|
|
|
{{<highlight text>}}
|
|
0;
|
|
1;
|
|
2;
|
|
{{< / highlight >}}
|
|
|
|
|
|
While not every browser may do this right, most browsers will render
|
|
the greyscale as its comes in, then add color to it one plane at a
|
|
time. It'll be more obvious over a slower connection; I purposely left
|
|
the image fairly large so that the transfer would be slower. You'll
|
|
note as well that the greyscale arrives much more slowly than the
|
|
color. (2020 note: most browsers will now let you use their
|
|
development tools to simulate a slow connection if you really want to
|
|
see.)
|
|
|
|
Code & Utilities
|
|
================
|
|
|
|
The **cjpeg** tool from libjpeg will (among other things) create a
|
|
JPEG using a custom scan script. Combined with ImageMagick, I used a
|
|
command like:
|
|
|
|
{{<highlight bash>}}
|
|
convert input.png ppm:- | cjpeg -quality 95 -optimize -scans scan_script > output.jpg
|
|
{{< / highlight >}}
|
|
|
|
|
|
Or if the input is already a JPEG, `jpegtran` will do the same
|
|
thing, losslessly (as it's merely reordering coefficients):
|
|
|
|
{{<highlight bash>}}
|
|
jpegtran -scans scan_script input.jpg > output.jpg
|
|
{{< / highlight >}}
|
|
|
|
|
|
libjpeg has some interesting features as well. Rather than decoding an
|
|
entire full-resolution JPEG and then scaling it down, for instance (a
|
|
common use case when generating thumbnails), you may set it up when
|
|
decoding so that it will simply do the reduction for you while
|
|
decoding. This takes less time and uses less memory compared with
|
|
getting the full decompressed version and resampling afterward.
|
|
|
|
The C code below (or [here](./jpeg_split.c) or this
|
|
[gist](https://gist.github.com/9220146)), based loosely on `example.c`
|
|
from libjpeg, will split up a multi-scan JPEG into a series of
|
|
numbered PPM files, each one containing a scan. Look for
|
|
`cinfo.scale_num` (circa lines 67, 68) to use the fast scaling
|
|
features mentioned in the last paragraph, and note that the code only
|
|
processes as much input JPEG as it needs for the next scan. (It needs
|
|
nothing special to build besides a functioning libjpeg. `gcc -ljpeg -o
|
|
jpeg_split.o jpeg_split.c` works for me.)
|
|
|
|
{{<highlight c>}}
|
|
// jpeg_split.c: Write each scan from a multi-scan/progressive JPEG.
|
|
// This is based loosely on example.c from libjpeg, and should require only
|
|
// libjpeg as a dependency (e.g. gcc -ljpeg -o jpeg_split.o jpeg_split.c).
|
|
#include <stdio.h>
|
|
#include <jerror.h>
|
|
#include "jpeglib.h"
|
|
#include <setjmp.h>
|
|
#include <string.h>
|
|
|
|
void read_scan(struct jpeg_decompress_struct * cinfo,
|
|
JSAMPARRAY buffer,
|
|
char * base_output);
|
|
int read_JPEG_file (char * filename, int scanNumber, char * base_output);
|
|
|
|
int main(int argc, char **argv) {
|
|
if (argc < 3) {
|
|
printf("Usage: %s <Input JPEG> <Output base name>\n", argv[0]);
|
|
printf("This reads in the progressive/multi-scan JPEG given and writes out each scan\n");
|
|
printf("to a separate PPM file, named with the scan number.\n");
|
|
return 1;
|
|
}
|
|
|
|
char * fname = argv[1];
|
|
char * out_base = argv[2];
|
|
read_JPEG_file(fname, 1, out_base);
|
|
return 0;
|
|
}
|
|
|
|
struct error_mgr {
|
|
struct jpeg_error_mgr pub;
|
|
jmp_buf setjmp_buffer;
|
|
};
|
|
|
|
METHODDEF(void) error_exit (j_common_ptr cinfo) {
|
|
struct error_mgr * err = (struct error_mgr *) cinfo->err;
|
|
(*cinfo->err->output_message) (cinfo);
|
|
longjmp(err->setjmp_buffer, 1);
|
|
}
|
|
|
|
int read_JPEG_file (char * filename, int scanNumber, char * base_output) {
|
|
struct jpeg_decompress_struct cinfo;
|
|
struct error_mgr jerr;
|
|
FILE * infile; /* source file */
|
|
JSAMPARRAY buffer; /* Output row buffer */
|
|
int row_stride; /* physical row width in output buffer */
|
|
|
|
if ((infile = fopen(filename, "rb")) == NULL) {
|
|
fprintf(stderr, "can't open %s\n", filename);
|
|
return 0;
|
|
}
|
|
|
|
// Set up the normal JPEG error routines, then override error_exit.
|
|
cinfo.err = jpeg_std_error(&jerr.pub);
|
|
jerr.pub.error_exit = error_exit;
|
|
// Establish the setjmp return context for error_exit to use:
|
|
if (setjmp(jerr.setjmp_buffer)) {
|
|
jpeg_destroy_decompress(&cinfo);
|
|
fclose(infile);
|
|
return 0;
|
|
}
|
|
jpeg_create_decompress(&cinfo);
|
|
jpeg_stdio_src(&cinfo, infile);
|
|
(void) jpeg_read_header(&cinfo, TRUE);
|
|
|
|
// Set some decompression parameters
|
|
|
|
// Incremental reading requires this flag:
|
|
cinfo.buffered_image = TRUE;
|
|
// To perform fast scaling in the output, set these:
|
|
cinfo.scale_num = 1;
|
|
cinfo.scale_denom = 1;
|
|
|
|
// Decompression begins...
|
|
(void) jpeg_start_decompress(&cinfo);
|
|
|
|
printf("JPEG is %s-scan\n", jpeg_has_multiple_scans(&cinfo) ? "multi" : "single");
|
|
printf("Outputting %ix%i\n", cinfo.output_width, cinfo.output_height);
|
|
|
|
// row_stride = JSAMPLEs per row in output buffer
|
|
row_stride = cinfo.output_width * cinfo.output_components;
|
|
// Make a one-row-high sample array that will go away when done with image
|
|
buffer = (*cinfo.mem->alloc_sarray)
|
|
((j_common_ptr) &cinfo, JPOOL_IMAGE, row_stride, 1);
|
|
|
|
// Start actually handling image data!
|
|
while(!jpeg_input_complete(&cinfo)) {
|
|
read_scan(&cinfo, buffer, base_output);
|
|
}
|
|
|
|
// Clean up.
|
|
(void) jpeg_finish_decompress(&cinfo);
|
|
jpeg_destroy_decompress(&cinfo);
|
|
fclose(infile);
|
|
|
|
if (jerr.pub.num_warnings) {
|
|
printf("libjpeg indicates %i warnings\n", jerr.pub.num_warnings);
|
|
}
|
|
}
|
|
|
|
void read_scan(struct jpeg_decompress_struct * cinfo,
|
|
JSAMPARRAY buffer,
|
|
char * base_output)
|
|
{
|
|
char out_name[1024];
|
|
FILE * outfile = NULL;
|
|
int scan_num = 0;
|
|
|
|
scan_num = cinfo->input_scan_number;
|
|
jpeg_start_output(cinfo, scan_num);
|
|
|
|
// Read up to the next scan.
|
|
int status;
|
|
do {
|
|
status = jpeg_consume_input(cinfo);
|
|
} while (status != JPEG_REACHED_SOS && status != JPEG_REACHED_EOI);
|
|
|
|
// Construct a filename & write PPM image header.
|
|
snprintf(out_name, 1024, "%s%i.ppm", base_output, scan_num);
|
|
if ((outfile = fopen(out_name, "wb")) == NULL) {
|
|
fprintf(stderr, "Can't open %s for writing!\n", out_name);
|
|
return;
|
|
}
|
|
fprintf(outfile, "P6\n%d %d\n255\n", cinfo->output_width, cinfo->output_height);
|
|
|
|
// Read each scanline into 'buffer' and write it to the PPM.
|
|
// (Note that libjpeg updates cinfo->output_scanline automatically)
|
|
while (cinfo->output_scanline < cinfo->output_height) {
|
|
jpeg_read_scanlines(cinfo, buffer, 1);
|
|
fwrite(buffer[0], cinfo->output_components, cinfo->output_width, outfile);
|
|
}
|
|
|
|
jpeg_finish_output(cinfo);
|
|
fclose(outfile);
|
|
}
|
|
{{< / highlight >}}
|
|
|
|
Examples
|
|
========
|
|
|
|
Here are all 10 scans from a standard progressive JPEG, separated out with the example code:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|