It's trivial to get 256 Bits and user Pio had the nice idea for 192 Bits: Store occupancy bitboard (64) + 32 * piece type (4) of the occupied squares in some deterministic order. This has the nice side effect that you can make it colour agnostic by having the order be depending on who to move.
A further improvement he came up with was Huffmann encoding the pieces:
Considering kings take 12 Bits here it should be denser to just take them out of the bitboard part and store their position as 6 Bit at a fixed position. A better way may be to have 1 Bit denote if they are on the starting position, and if they are you can use those spare 6 Bits to denote their castling rights.King positions will need 12 bits Pawns will be encoded as 1, knights as 011, bishops as 010, rooks as 001 and queen as 000.
These changes together probably save at least some 20 Bits (things get a bit tricky when pawns promote into bigger pieces; the first promotion will require a capture so nothing would break there however the second promotion may not have a capture; so if you want to be save you probably have to give 16 Bit of those saved 32 back to ensure no problems for some exotic positions; or one may use the trick further below)
There is some fiddling to do with e.p. too but if smartly done that may not take any more space.
You then will want to get 20 or so more bits saved by choosing the bucket, the simplest here would be to just fold it through XOR. This may not give a nice distribution though the hope could be that the Huffmann compression and possibly some future ideas might at least help scramble the data a bit. If not you'd have to throw an additional hash function on it I suppose.
With 40 saved Bits you would have quite a bit of space for the perft count or for the search information. However in the perft case some select few positions may need more than 40 Bits. So additionally one could allocate one Bit to tell if this is a 192 or 256 Bit hash entry. In the latter case you would have enough space for the hash code of exotic positions and for a high perft count.
You would then have a bucket be 4 * 192 = 3 * 256 Bit = 92 Bytes. Might not be ideal for hash line length but you can probably use some smart prefetching to make that not make a difference.
Overall a TT entry in this usually costs 24 Byte here. This is more expensive than usual however not *that* much more expensive, and you do get colour agnostic which may help in some opening positions for search or with the perft efficiency depending on the position.
Generating the hash code is quite a bit more expensive that a Zobrist hash. One might still try to make it somewhat incremental though that will be a bit of a headache... On the other hand in the search you might (?) save yourself the move validation part if you use lockless hashing?
While I don't think this beats Zobrist hashing methods in terms of play strength it might be an interesting idea to think about and maybe test.
Thoughts? Any good ways to squeeze out another few Bits. (I mean, there have to be; you could for example get something out of the occupancy bitboard as you can have at most 32 1s, 30 without the kings; though compressing that may not be so trivial)