The highcolor layer is a special layer which reuses data from the four tilemap layers (it can be used at the same time as the tilemaps, and even blend with them, but it is not useful to do so).
Effectively, it is a 32x32 (or 64x32 in extend mode) tile layer, where each tile is a 4×16 texture in rom with up to 255(?) test this, but it's definitely a lot. in theory, 16 bits per pixel colors (plus transparency). it cannot be scrolled or scaled horizontally.
as far as we know, no games have used this feature, but i have tested it enough to know that it works.
explain fully how the texture data is sourced. it's annoyingly low level, but kinda necessary in order to understand how all the tilemap settings affect the highcolor layer.
Line Register 2.0 "6000"
#,#,#,#,#,#,#,h,#:8
h
Highcolor Layer: Alpha Select
Line Register 2.2 "6400"
#:2,#,#,#,#,h,#,#:4,#:4
h
Highcolor Layer: Mosaic Enable
Line Register 3.0 "7000"
bm:2,clip:10,prio:4
prio
Highcolor Layer: Priority (0-15)
clip
Highcolor Layer: Clip Settings
bm
Highcolor Layer: Blend Mode
0
both
1
A
2
B
3
neither
seems to behave differently in FDA mode 2. but it would be really strange for that setting to affect this, so we need to test more to be sure.