Previous: A Simple Application for Color Naming, Pointing Out, and Selecting Up: An Application and Some Empirical Results
The best way to get a good feel for how well (or how poorly) the algorithm
performs is to run it for oneself (See Appendix for downloading
instructions). Barring that, the best we can do is look at an image and the
judgments the algorithm makes on it. Again, I must caution against taking
the color reproduction on paper too seriously; too many unknowns are
involved, but it will do for a rough impression. Figure
shows a particular test image (which has not been involved in ``training''
or optimizing the category models at all) taken with a hand-held regular
home camcorder whose composite NTSC
signal
was hooked up to a cheap Sun VideoPix frame grabber. This equipment is
certainly not of lab grade quality, not calibrated, and the lighting for
the scene was just the available office TL lighting, not controlled or
adjusted in any way. The image shows some toy utensils and other objects of
various colors, none particularly good examples of Basic (or primary)
colors.
A trace of the run is shown below, with only minor editing to remove superfluous white space, and with comments added, preceded by percent signs (``%'').
% the demo1 function starts the external image display and blob % selection/display program, and calls the various functions in turn with % appropriate parameters.In[2]:= demo1[] 520 x 381 pixels, max value 255 Color Naming
Mouse commands: left - sample blob around point middle - stop sampling right - toggle examples display Keyboard commands (in the Color Naming window): S - Subsample and save Q - Quit program
% The first thing it does is to call the adaptation function, which needs % to read (a subsampled version of) the image. Since the display program % and the Mathematica process communicate asynchronously, we see their % respective outputs interleaved.
% this is Mma
Please sample the image working...
% this is the display prg
Wrote 43 x 23 subsampled file /projects/lammens/pix/subsampled.ppm
% This is the adaptation function reporting the normalized RGB values found % for black and white, as described above.
{BlackRGB, WhiteRGB} = {{0.196078, 0.203922, 0.164706},
> {0.937255, 0.945098, 0.988235}}
% Next the color naming function is called, in simple name mode. It uses a % category membership threshold of 0.35, and the L*a*b* color space. It asks % for a sample which it gets from the display/choose program, which in turn % reports the coordinates of the center of the blob and the average % unnormalized RGB values. This corresponds to the thin rectangle labeled % ``1'' in the figure (upper left corner). Note that the RGB values are % around 50 each, or 0.25 normalized, i.e. not black in absolute terms.
*** Color Naming, simple names *** sample? Blob at (8,9): 51 54 46
% After turning the crank a bit, the naming function returns its color % judgment as a list of {name, goodness} pairs (only one has been % requested). The reported color is black, because of the adaptation % performed. It's not all that good a black, however.
{{black, 0.573432}}
% Sample 2 selects the highlight on the blade of the knife, which is a % specular reflection of the overhead office lights. Not surprisingly, the % answer is ``white'' -- remember that these algorithms work in a % context-free manner. It's a pretty good white too.
sample? Blob at (417,192): 252 253 254 {{white, 0.876566}}
% Sample 3 is from the white paper background, in the upper right quadrant. % This too is recognized as white, but a less good one than the previous % sample.
sample? Blob at (480,79): 223 253 249 {{white, 0.657369}}
% Sample 4 is again from the white paper background, but a slightly more % shaded area near the middle. Again the answer is white, but a less good % one still.
sample? Blob at (240,96): 197 227 220 {{white, 0.55741}}
% Sample 5 is in the shadow of the bowl, and in these case a ``null'' % judgment is returned, meaning the sample does not belong to any basic % color category. Here our visual system's context-sensitive operation is % clearly superior (but try looking at the sample through a hole in a piece % of white paper, and see if it still looks white).
sample? Blob at (200,95): 164 189 163 {}
% Sample 6 is from the left middle piece of desktop showing underneath the % white paper, and it is reported as a pretty good gray. The human visual % system would connect this sample with the one from the upper left corner % and not report them as different in color.
sample? Blob at (7,236): 74 86 77 {{gray, 0.878559}}
% Sample 7 is from the upper right bottom of the bowl, and is % reported as a not very good green (in fact barely above the threshold for % category memberhip of 0.35)
sample? Blob at (140,87): 183 225 33 {{green, 0.38532}}
% Sample 8 is from the shaded lower left inner wall of the bowl, but is % nevertheless reported as a virtually equally good (or bad) green. The % algorithm thus show some measure of robustness with respect to varying % lighting.
sample? Blob at (62,143): 134 164 11 {{green, 0.385104}}
% Sample 9 is from the lower region of the bowl where the wall and the % bottom join, and is reported as a moderate yellow. The % color seems to be in between yellow and green, according to the % algorithm. And again there are no constraints on neighboring samples to % be judged the same, so the category may suddenly switch due to small % local variations.
sample? Blob at (93,153): 135 162 16 {{yellow, 0.451289}}
% Sample 10 is from the upper middle part of the plate, and reported as a % moderate blue
sample? Blob at (280,202): 119 189 194 {{blue, 0.418403}}
% Sample 11 is from the lower left part of the plate, and reported as a % virtually identical blue.
sample? Blob at (236,280): 112 179 178 {{blue, 0.413634}}
% Sample 12 is from the left part of the plate, and contains a rather % bright highlight, but the reported color and goodness are almost the same % anyway.
sample? Blob at (202,247): 127 190 193 {{blue, 0.403172}}
% Sample 13 is from the spoon handle, and reported as a moderate blue too. % This is not entirely satisfactory, but see below.
sample? Blob at (142,299): 115 99 162 {{blue, 0.420509}}
% Samples 14 and 15 are from the fork and knife handles, and reported the % practically the same color as the spoon, notwithstanding the difference % in lighting.
sample? Blob at (463,301): 147 125 219 {{blue, 0.423843}}
sample? Blob at (413,260): 146 127 201 {{blue, 0.412522}}
% Sample 16 is from the upper right of the inner wall of the cup, and % reported as a moderately good gray. This is again not satisfactory.
sample? Blob at (393,57): 118 77 73 {{gray, 0.446365}}
% Sample 17 is from a different part of the cup, containing a highlight, % but it also reported as gray.
sample? Blob at (347,73): 191 139 140 {{gray, 0.415984}}
% A null sample signals the end of the current naming mode, and the next % function called is again the naming function, but this time in complex % name mode. The samples are the same as the first 17 (plus or minus a few % pixels, because they were entered anew with the mouse).
sample? *** Color Naming, complex names ***
% Sample 18. The earlier ``black'' judgment is now refined to a somewhat % greenish-black. The modifier (secondary category) is named first, and the % goodness values are given in the same order. The algorithm currently does % not provide a single goodness value for the compound name.
sample? Blob at (7,9): 51 54 46 {somewhat green-ish black, {0.343353, 0.573432}}
% Sample 19. The white highlight is now reported as somewhat pinkish white, % probably due to the colored substrate of the knife (pink is relatively % close to purple in the color space).
sample? Blob at (417,193): 252 253 255 {somewhat pink-ish white, {0.324792, 0.884955}}
% Samples 20 and 21. The background becomes blue-ish white, whish is rather % accurate (compare to the white of the paper surrounding the figure). Note % that the white goodness goes down, and the blue goodness up a bit as we % move into the more shaded regions of the white paper.
sample? Blob at (488,79): 223 253 249 {somewhat blue-ish white, {0.303342, 0.657369}}
sample? Blob at (241,95): 196 227 219 {somewhat blue-ish white, {0.315056, 0.530577}}
% Sample 22. The heavily shaded region is still not in any basic category.
sample? Blob at (200,94): 164 189 163 none
% Sample 23. The gray of the desk top is now reported as bluish, which is % odd.
sample? Blob at (7,237): 74 86 77 {blue-ish gray, {0.355959, 0.878559}}
% Sample 24. The green of the bowl bottom is still unqualified green, % albeit not a very good one.
sample? Blob at (138,85): 181 222 33 {green, 0.386859}
% Sample 25. Elswhere in the bowl, yellow creeps in.
sample? Blob at (56,140): 135 161 11 {somewhat yellow-ish green, {0.340837, 0.379807}}
% Sample 26. When the top candidate changes, the previous top candidate now % becomes the modifier, indicating some waffling on the part of the naming % algorithm.
sample? Blob at (89,150): 135 164 16 {green-ish yellow, {0.426267, 0.442337}}
% Samples 27-29. The plate is now reported as grayish blue, which is quite % acceptable.
sample? Blob at (275,197): 120 188 192 {somewhat gray-ish blue, {0.263667, 0.414681}}
sample? Blob at (232,279): 112 178 178 {somewhat gray-ish blue, {0.301262, 0.415194}}
sample? Blob at (203,247): 127 190 193 {somewhat gray-ish blue, {0.269991, 0.403172}}
% Samples 30-32. The cutlery is now reported as either blue or pinkish % blue, which is not bad, but purple or purplish blue would be more % appropriate.
sample? Blob at (141,301): 116 100 163 {blue, 0.420315}
sample? Blob at (464,294): 148 125 218 {somewhat pink-ish blue, {0.273953, 0.420837}}
sample? Blob at (414,259): 151 134 206 {somewhat pink-ish blue, {0.293498, 0.410624}}
% Samples 33-34. The cup color is now modified to pinkish gray, which is % reasonable given the highlights, but not great.
sample? Blob at (392,58): 119 77 73 {somewhat pink-ish gray, {0.289888, 0.434842}}
sample? Blob at (346,73): 186 135 135 {pink-ish gray, {0.372527, 0.440735}}
% After a null sample, the next function called is the pointing-out % function. It is asked to report the best 3 examples of every color given % to it by name, but only the top choice is shown in the figure, to reduce % clutter. The threshold used is now much lower at 0.01, to encourage the % algorithm to be maximally cooperative and broad-minded (broad-categoried, % actually). Before it can point anything out, it needs to (sub)sample the % image, which is effectively its domain of discourse, or its environment, % for this task.
sample? *** Pointing Out Colors, simple or complex names *** Please sample the image Wrote 43 x 23 subsampled file /projects/lammens/pix/subsampled.ppm working... Enter color names as quoted [list of] string[s], or "" to stop
% First we enter only simple color names. The returned result is a list of % the top three candidate referents in the image, specified as triplets of % internal color space coordinates, goodness value, and linearized index. % The index is converted back to (x,y) image coordinates and passed to the % pointing/display program, which draws boxes of varying width around the % corresponding image blobs. The first one of these is shown as a thick % rectangle in the figure, with the corresponding abbreviated color name. % The ``>'' signs in the margin are Mathematica's continuation sign for % multi-line outputs.
% The best example of black in the image is the blob in the upper left % corner, but it's not a very good one.
Color: "black" {{{-0.949934, -8.93606, 8.94645}, 0.573432, 0}, {{0., 0., 0.}, 0.499995, 43},
> {{-8.80682, -0.988521, 16.7996}, 0.434245, 1}}
% The best example of white is the highlight on the knife blade.
Color: "white" {{{0., 0., 100.}, 0.994671, 550}, > {{1.19341, -2.5512, 97.4658}, 0.925709, 507}, > {{-3.30085, 1.8013, 98.0898}, 0.896158, 334}}
% The best example of gray is the blob halfway down the left edge. This is % probably the only one that does not contain part of the paper edge, other % than the ones in the upper left corner.
Color: "gray" {{{-7.9972, 3.37291, 52.6723}, 0.912785, 645}, > {{-9.15905, 2.19436, 54.5701}, 0.906193, 688}, > {{3.36029, -6.16337, 62.3559}, 0.898141, 484}}
% The best example of red is on the inner wall of the cup, but its goodness % value is very low. The redness does not escape the program's attention, % so to speak, but the goodness is not high enough to label the object as % red in the naming mode.
Color: "red" {{{26.9112, 14.3771, 40.8237}, 0.0651646, 290}, > {{24.6857, 10.9119, 46.7973}, 0.0347762, 248}, > {{24.8426, 10.6163, 53.0361}, 0.0235578, 160}}
% As expected, the best exmamples of both green and yellow are from the bowl.
Color: "green" {{{-33.8928, 119.015, 78.373}, 0.443789, 91}, > {{-33.0592, 128.975, 77.1126}, 0.442608, 175}, > {{-29.7397, 128.662, 75.3204}, 0.441054, 437}}
Color: "yellow" {{{-30.7869, 148.075, 74.873}, 0.471412, 438}, > {{-29.7397, 128.662, 75.3204}, 0.4484, 437}, > {{-32.0185, 149.603, 76.4553}, 0.446844, 394}}
% The best blue is from the plate
Color: "blue" {{{-20.6264, -5.98904, 77.6899}, 0.428033, 532}, > {{-21.6781, -5.23694, 76.3669}, 0.426202, 926}, > {{12.1377, -21.1522, 62.9835}, 0.425152, 829}}
% The best brown is in the upper left corner, from the desk top, but the % second best canidate (not shown in the figure) is actually on the cup, % which is good (brown is a kind of dark red, in an artist's conception of % color).
Color: "brown" {{{0., 0., 0.}, 0.371792, 43}, {{26.9112, 14.3771, 40.8237}, 0.318304, 290}, > {{-0.949934, -8.93606, 8.94645}, 0.269128, 0}}
% The best purple is appropriately on the fork. So again, even if a % different category is chosen in the naming mode, the purpleness is noted % nevertheless.
Color: "purple" {{{22.1124, -29.2723, 68.7999}, 0.0935326, 640}, > {{26.9112, 14.3771, 40.8237}, 0.09335, 290}, > {{22.4642, -30.4113, 71.309}, 0.0923015, 894}}
% The best pink is appropriately from the cup highlight
Color: "pink" {{{16.3689, 3.61084, 78.632}, 0.385165, 201}, > {{17.8138, 5.53527, 70.6242}, 0.354974, 243}, > {{20.2324, 4.45821, 66.0619}, 0.348803, 202}}
% The best orange is also on the cup, but the goodness is extremely low, % almost below the already very low threshold. Only one above-threshold % example is found in this case.
Color: "orange" {{{26.9112, 14.3771, 40.8237}, 0.0126883, 290}}
% Interestingly, the best example of grayish blue is on the plate, but the % best example of bluish gray is on the desk top, demonstrating that % compound category names do not simply represent a conjunction of the two % categories.
Color: {"gray", "blue"} {{{-20.6264, -5.98904, 77.6899}, {0.428033, 0.348169}, 532}, > {{-21.6781, -5.23694, 76.3669}, {0.426202, 0.356378}, 926}, > {{18.9681, -27.0463, 63.0806}, {0.424739, 0.281654}, 742}}
Color: {"blue", "gray"} {{{-7.9972, 3.37291, 52.6723}, {0.912785, 0.337975}, 645}, > {{-9.15905, 2.19436, 54.5701}, {0.906193, 0.349694}, 688}, > {{3.36029, -6.16337, 62.3559}, {0.898141, 0.36831}, 484}}
% The best examples of yellowish green and greenish yellow are both on the % bowl however, again indicating the ambiguity of the color in the % algorithm's ``mind''.
Color: {"yellow", "green"} {{{-33.8928, 119.015, 78.373}, {0.443789, 0.358894}, 91}, > {{-33.0592, 128.975, 77.1126}, {0.442608, 0.409575}, 175}, > {{-32.3242, 110.273, 78.338}, {0.441002, 0.329341}, 306}}
Color: {"green", "yellow"} {{{-30.7869, 148.075, 74.873}, {0.471412, 0.429748}, 438}, > {{-29.7397, 128.662, 75.3204}, {0.4484, 0.441054}, 437}, > {{-32.0185, 149.603, 76.4553}, {0.446844, 0.425336}, 394}}
% After a null name, the next function activated is the one to select % objects by (simple) color names. A number of object samples are collected % first, representing each of the objects in the image as well as the % paper background and the desk top.
Color: "" *** Choosing Objects by Color, simple names *** object sample? Blob at (109,104): 165 205 25 object sample? Blob at (137,205): 130 114 176 object sample? Blob at (276,251): 119 186 189 object sample? Blob at (417,294): 147 125 214 object sample? Blob at (465,274): 148 127 213 object sample? Blob at (376,46): 120 77 75 object sample? Blob at (389,153): 228 254 251 object sample? Blob at (9,11): 50 54 45 object sample? Enter color names as quoted string[s], or "" to stop
% The circles in the figure represent the object samples, and the % abbreviated color names next to them represent the choice the algorithm % makes if given the corresponding color name. The same cooperative % threshold of 0.01 is used.
% Black and white lead to the expected choices.
Color: "black" {{{-7.88055, -6.83839, 7.5829}, 0.458093, 7}}
Color: "white" {{{-6.21646, 2.01139, 101.314}, 0.710922, 6}}
% When given a limited number of ojbects to choose from, gray, red, purple, % and pink all refer to the same object, viz. the cup.
Color: "gray" {{{26.7022, 6.30509, 50.5676}, 0.425217, 5}}
Color: "red" {{{26.7022, 6.30509, 50.5676}, 0.0226092, 5}}
Color: "purple" {{{26.7022, 6.30509, 50.5676}, 0.0974246, 5}}
Color: "pink" {{{26.7022, 6.30509, 50.5676}, 0.297329, 5}}
% As expected, yellow and green both refer to the bowl
Color: "green" {{{-33.6247, 99.8405, 87.0685}, 0.408762, 0}}
Color: "yellow" {{{-33.6247, 99.8405, 87.0685}, 0.222119, 0}}
% Blue refers to the knife, with the fork and spoon probably close % runners-up
Color: "blue" {{{21.2171, -29.1064, 72.5961}, 0.420029, 3}}
% brown refers to the desk top
Color: "brown" {{{-7.88055, -6.83839, 7.5829}, 0.25868, 7}}
% orange does not refer to anything
Color: "orange" huh?
% None of the simple color names refers to the plate. Probably that would % need a complex color name, but this has not been implemented for the % selection task.
Color: ""
% To get an idea of the effectiveness of the adaptation algorithm, we now % run the some of the same tests without it, using the default absolute % black and white values. For brevity, only the results that differ from % the previous ones are commented. These results are not shown in the figure.
In[4]:= demo2[] 520 x 381 pixels, max value 255 Color Naming
Mouse commands: left - sample blob around point middle - stop sampling right - toggle examples display Keyboard commands (in the Color Naming window): S - Subsample and save Q - Quit program
*** Without Adaptation *** {BlackRGB, WhiteRGB} = {{0, 0, 0}, {1, 1, 1}} *** Color Naming, simple names ***
% Sample 1 (desk) was black before, but is now gray.
sample? Blob at (9,9): 51 55 46 {{gray, 0.95517}} sample? Blob at (417,192): 252 253 254 {{white, 0.997448}} sample? Blob at (477,84): 222 252 248 {{white, 0.827816}} sample? Blob at (242,93): 195 226 217 {{white, 0.643781}} sample? Blob at (200,93): 164 188 163 {} sample? Blob at (6,237): 75 86 76 {{gray, 0.890602}}
% Sample 7 (bowl) was green before, and is now nothing.
sample? Blob at (138,84): 180 222 33 {} sample? Blob at (59,142): 134 163 11 {{green, 0.382396}}
% Sample 9 (bowl) was yellow before, and is now green
sample? Blob at (92,151): 135 163 16 {{green, 0.375072}} sample? Blob at (277,198): 120 189 193 {{blue, 0.396411}} sample? Blob at (236,284): 112 179 177 {{blue, 0.391728}} sample? Blob at (202,249): 127 189 192 {{blue, 0.387349}}
% Sample 13 (spoon handle) was blue before, and is now gray.
sample? Blob at (141,304): 116 100 163 {{gray, 0.443478}} sample? Blob at (465,296): 148 126 219 {{blue, 0.413124}} sample? Blob at (413,258): 150 134 203 {{blue, 0.397054}} sample? Blob at (392,59): 119 77 72 {{gray, 0.734896}} sample? Blob at (347,71): 186 134 135 {{gray, 0.432342}} sample? *** Color Naming, complex names ***
% sample 18 (desk) was somewhat greenish black before, and is now somewhat % blueish gray
sample? Blob at (9,9): 51 55 46 {somewhat blue-ish gray, {0.318699, 0.95517}} sample? Blob at (417,191): 252 252 254 {somewhat pink-ish white, {0.32611, 0.997364}} sample? Blob at (483,77): 223 254 250 {somewhat blue-ish white, {0.319529, 0.821147}} sample? Blob at (237,94): 198 228 219 {somewhat blue-ish white, {0.323538, 0.672593}} sample? Blob at (199,94): 164 189 162 none
% Sample 23 (desk) was blueish gray before, and somewhat blueish gray now
sample? Blob at (8,237): 74 86 77 {somewhat blue-ish gray, {0.34003, 0.892704}}
% Sample 24 (bowl) was green before, none now
sample? Blob at (136,86): 179 221 32 none
% Sample 25 was somewhat yellow-ish green before, green now
sample? Blob at (61,140): 136 165 11 {green, 0.381509}
% Sample 26 was green-ish yellow before, green now.
sample? Blob at (89,148): 135 164 16 {green, 0.375411} sample? Blob at (282,199): 120 190 194 {somewhat gray-ish blue, {0.277526, 0.396891}} sample? Blob at (233,277): 112 178 178 {somewhat gray-ish blue, {0.323495, 0.394109}} sample? Blob at (202,246): 127 190 193 {somewhat gray-ish blue, {0.285553, 0.387858}}
% Sample 30 was blue before, blue-ish gray now
sample? Blob at (141,302): 116 100 163 {blue-ish gray, {0.406485, 0.443478}} sample? Blob at (463,299): 147 125 219 {somewhat pink-ish blue, {0.297212, 0.414387}}
% Sample 32 was somewhat pink-ish blue before, gray-ish blue now
sample? Blob at (412,260): 142 125 195 {gray-ish blue, {0.350279, 0.400874}} sample? Blob at (388,52): 119 77 75 {somewhat pink-ish gray, {0.291203, 0.739966}} sample? Blob at (347,72): 188 137 138 {pink-ish gray, {0.352761, 0.421948}} sample? *** Pointing Out Colors, simple or complex names *** Please sample the image Wrote 43 x 23 subsampled file /projects/lammens/pix/subsampled.ppm working... Enter color names as quoted [list of] string[s], or "" to stop
% The previous best example of black was on the desk, now there is none.
Color: "black" {} Color: "white" {{{0.532836, -2.37186, 97.8842}, 0.947593, 550}, > {{-2.09951, -0.735345, 96.4016}, 0.914406, 334}, > {{-2.68196, -2.32448, 97.9623}, 0.904322, 552}}
% The previous best example of gray was half way down the left edge, now it % is in the upper left.
Color: "gray" {{{-3.02248, 4.28535, 52.6059}, 0.961785, 0}, > {{-4.55361, 3.84577, 55.9149}, 0.961349, 44}, > {{-4.38929, 4.59856, 55.5826}, 0.956423, 86}}
% now no best example of red is found
Color: "red" {}
% the best examples of yellow and green are still from the bowl, but in % different places than before
Color: "green" {{{-23.5407, 62.7201, 80.1716}, 0.381321, 348}, > {{-23.3872, 61.536, 79.7746}, 0.380848, 392}, > {{-24.859, 61.5938, 81.5351}, 0.378426, 305}} Color: "yellow" {{{-23.5407, 62.7201, 80.1716}, 0.110165, 348}, > {{-23.3872, 61.536, 79.7746}, 0.106706, 392}, > {{-25.7179, 63.496, 83.541}, 0.102021, 398}}
% The best blue was on the knife handle before, now on the plate
Color: "blue" {{{14.5833, -22.6702, 77.6263}, 0.413639, 851}, > {{14.9457, -23.0869, 78.1905}, 0.413597, 812}, > {{15.2259, -22.919, 77.2831}, 0.41233, 894}} Color: "brown" {{{-3.09103, 5.85521, 51.7689}, 0.147524, 43}, > {{-3.02248, 4.28535, 52.6059}, 0.138768, 0}, > {{9.80109, 9.4391, 60.507}, 0.138071, 290}} Color: "purple" {{{15.2259, -22.919, 77.2831}, 0.0661223, 894}, > {{14.5268, -21.4402, 75.6475}, 0.0654806, 640}, > {{14.9457, -23.0869, 78.1905}, 0.0647991, 812}} Color: "pink" {{{11.0969, 2.14573, 82.0917}, 0.360105, 201}, > {{11.043, 4.11308, 76.6872}, 0.336285, 243}, > {{11.9717, 3.68921, 73.7798}, 0.330684, 202}}
% the best orange was on the cup before, now none is found
Color: "orange" {} Color: ""
Some tests on rather different images, for instance color cartoons acquired with a scanner, show very comparable results. Although the application presented in this chapter is an interesting demonstration of what can be accomplished with the color perception and naming model, it is certainly open for improvement. For instance, it would be interesting to explore an active version of the adaptation algorithm that could move the camera around searching for appropriate lighting conditions.