BIZNIQUE / LINEAR ALGEBRA / 27
Balance encoding size and useful structure
Balance encoding size and useful structure
Representation changes the shape of the data as well as the information a model can use.
Embedded Files
THE IDEA
Read the example.
Read the example.
A categorical feature with d known categories can require d one-hot columns. The representation is often sparse because one category activates only one indicator. Dense and sparse storage have very different costs.
One-hot width grows with category count
Use it well
Use it well
Hashing introduces possible collisions; embeddings introduce learned dense parameters. Neither is automatically preferable. Evaluate memory, latency, interpretation and task quality together.
TRY IT YOURSELF
A quick check.
A quick check.
Why can a wide one-hot matrix remain small in sparse form?
Check your reasoning
Check your reasoning
Most entries are implicit zeros, so the stored values and indices can scale with active entries rather than all possible entries.
Page updated
Google Sites
Report abuse