Shadow

Shadow mode takes effect when there are two layers with neighboring priority values. (within the halfpixel column)explain this better, maybe we need to define the pipeline elsewhere and then we can just say this stuff.  Anywhere both layers overlap (have nontransparent pixels), the upper layer's color is altered.  this is the only situation where a pixel can have an effect on the output without being the highest priority in its halfpixel.  The effect varies based on the bpp setting of the upper layer's pixel.  Generally: in shadow mode 2, it sets the lowest bit in the palette index to 1, and in mode 3 (never used), it sets most of the palette bits to 0

t = texture bit (n)
p = palette bit (n-4)
Q = palette bit (n-4) OR'd with texture bit (n)
0 = always 0
1 = always 1
(note that palette and texture bits are never shifted around)

bpp=0
 shadow=0 [p pppp pppp tttt]
 shadow=2 [p pppp ppp1 tttt]
 shadow=3 [0 0000 00pp tttt]
bpp=1
 shadow=0 [p pppp pppQ tttt]
 shadow=2 [p pppp pp1Q tttt]
 shadow=3 [0 0000 00p0 tttt]
bpp=2
 shadow=0 [p pppp ppQp tttt]
 shadow=2 [p pppp ppQp tttt] (no change)
 shadow=3 [0 0000 000p tttt]
bpp=3
 shadow=0 [p pppp ppQQ tttt]
 shadow=2 [p pppp p1QQ tttt]
 shadow=3 [0 0000 0000 tttt]

EXCEPT, with the pivot layer, it's like:
 shadow=0 [0 00pp pppp tttt]
 shadow=2 [0 00pp ppp1 tttt]
 shadow=3 [1 0000 0000 tttt]

note that sprites normally have color bit 12 hardcoded to 1, which is uniquely cleared by shadow mode 3

Shadow only works between certain combinations of upper/lower layers, though all layers can affect the results via priority conflicts (e.g. normally sprites can't shadow onto tilemaps, but you can shadow a tilemap onto another tilemap, and use conflicting sprites to cut a hole in the lower tilemap, which will cut a hole in the shadow effect)  There isn't any clear rule for which layers are compatible: