When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers.
As a bonus, any operation can be replaced with a lookup into a nxn table.
In standard FP32, the infs are represented as a sign bit, all exponent bits=1, and all mantissa bits=0. The NaNs are represented as a sign bit, all exponent bits=1, and the mantissa is non-zero. If you used that interpretation with FP4, you'd get the table below, which restricts the representable range to +/- 3, and it feels less useful to me. If you're using FP4 you probably are space optimized and don't want to waste a quarter of your possible combinations on things that aren't actually numbers, and you'd likely focus your efforts on writing code that didn't need to represent inf and NaN.
You need it if you want the idea of total ordering over the extended Reals. There's +/- infinity--an affine closure, not projective (point at infinity)--so to make that math work you need to give 0 a sign.
That sounds pretty niche. What's a use case where you have less than 8 bits and that distinction is more important than having an extra finite value? I don't think AI is one.
For neural net gradient descent, automatic differentiation etc, the widely used ReLU function has infornation carrying derivatives at +0 and –0 if those are infinitesimals.
Barely any information. After surviving RELU that signed zero is probably getting added to another value and then oops the information is gone. It sounds a lot worse than properly spaced values.
If you were looking at the entire number line, sign would roughly be the most important part.
But you still have all the other numbers carrying sign info. This is only the sign of denormals and that's way less valuable. Outside of particular equations it ends up added to something else and disappearing entirely. It would be way better to cut it and have either half the smallest existing positive value or double the largest existing value as a replacement. Or many other options.
For FP4, yes... sometimes... it depends. But newer Nvidia architecture eg Blackwell w/ NVFP4 does not, they perform micro block scaling in the core. For older architectures, low quants like FP4 are also often not done native, and instead inflated back to BF16, eg with BnB.
As explained in an article linked at the bottom of TFA, the weights of a LLM have a normal (Gaussian) distribution.
Because of that, the best compromise when the weights are quantized to few levels is to place the points encoded by the numeric format used for the weights using a Gaussian function, instead of placing them uniformly on a logarithmic scale, like the usual floating-point formats attempt.
In fact it's not. The name itself mimicks cat, not less. It's a filter that adds annotations to its input, such as syntax highlighting, git diffs and special-char coloring.
Personally I can't find any use for bat: I'm a devote user of vim for editing, and it already does all of this, so why not using it to view files as well? It's satisfying to have the same interface, colors and shortcuts whether you're editing or viewing!
I feel that saying that EML can't generate all the elementary functions because it can't express the solution of the quintic is like saying that NAND gates can't be the basis of modern computing because they can't be used to solve Turing's halting problem.
As is usual with these kinds of "structure theorems" (as they're often called), we need to precisely define what set of things we seek to express.
A function which solves a quintic is reasonably ordinary. We can readily compute it to arbitrary precision using any number of methods, just as we can do with square roots or cosines. Not just the quintic, but any polynomial with rational coefficients can be solved. But the solutions can't be expressed with a finite number of draws from a small repertoire of functions like {+, -, *, /}.
So the question is, does admitting a new function into our "repertoire" allow us to express new things? That's what a structure theorem might tell us.
The blog post is exploring this question: Does a repertoire of just the EML function, which has been shown by the original author to be able to express a great variety of functions (like + or cosine or ...) also allow us to express polynomial roots?
> If you don't manage the history properly in your SPA, pressing the back button could take the user out of the app entirely.
Yes. And that should be the default behavior: browser buttons should take you through the browser's history. If you keep a in-app state and want the user to navigate through it, you should provide in-app buttons.
Nobody complains that the browser's close button quits the browser instead of the app it's showing, or that the computer's power button shuts down the whole OS and not only the program in the foreground.
Users must be educated. If they have learned that left means "back" and right means "forward", that a star (sometimes a heart) means "remember this for me", and that an underlined checkmark means "download", then understanding the concept of encapsulation shouldn't be too much for them.
> Yes. And that should be the default behavior: browser buttons should take you through the browser's history. If you keep a in-app state and want the user to navigate through it, you should provide in-app buttons.
The Back and Forward buttons on a web browser is the navigation for the web. If you click a link on a static html page it will create a new entry. If you click back, it'll take you back. If you press forward, You will navigate forward.
We should not be creating a secondary set of controls that does the same thing. This is bad UX, bad design, and bad for an accessible web.
> Nobody complains that the browser's close button quits the browser instead of the app it's showing, or that the computer's power button shuts down the whole OS and not only the program in the foreground.
It does close the app it's showing because we have tabs. If you close a tab, it'll close the app that it's showing. If you close the browser, which is made up of many tabs, it closes all of the tabs. Before tabs, if you closed a window, the web page you were on would close as well. It does what is reasonably expected.
If on your web application you have a 'link' to another 'page' where it shows a change in the view, then you'd expect you would be able to press back to go back to what you were just looking at. SPAs that DON'T do that are the ones that are doing a disservice to the user and reasonable navigation expectations.
> Users must be educated. If they have learned that left means "back" and right means "forward", that a star (sometimes a heart) means "remember this for me", and that an underlined checkmark means "download", then understanding the concept of encapsulation shouldn't be too much for them.
They should not have to be 'educated' here. The mental model of using the back and forward buttons to navigate within a webpage is totally fine.
Sorry, I've read and reread TFA but the concept still evades me. Is it that, since it's easier for a hash function to have higher entropy in the higher bits than in the lower ones, it would be more logical for hash tables to discard the lower bits and keep the higher ones?
Higher entropy in the higher bits is a property of addition and multiplication when the result has the same size as the operands (i.e. the result is taken modulo 2^N, which folds back the values exceeding the size of the result).
When other operations are used, there may be other bits with higher entropy. For example, when full-length multiplication is used (i.e. where the length of the result is the sum of the lengths of the operands), the middle bits have the highest entropy, not the top bits. On CPUs like the Intel/AMD x86-64 CPUs, where fast long multiplication instructions are available, this can be exploited in more performant hash functions and PRNGs.
In hash functions, additions and multiplications are frequently used together with rotations, in order to redistribute the entropy from the top bits to the bottom bits, during the following operations.
Honorary mention: byte swapping instructions (originally added to CPUs for endianness conversion) can also be used to redistribute entropy, but they're slightly slower than rotations on Intel, which is why I think they aren't utilized much.
Yes, that's my point. It's not true that all hash functions have this characteristic, but most fast ones do. (And if you're using a slow-and-high-quality hash function, the distinction doesn't matter, so might as well use top bits.)
Oh wait...
reply