Quadrillion-parameter transformers and how to train them
30 points by frozenseven 6 days ago | 17 comments
notlibrary 28 minutes ago
The planet has 20 quadrillions of ants
replysmokel 13 minutes ago
And several thousand quadrillon grains of sand.
replyintrasight 2 minutes ago
There are more ways to shuffle a deck of cards than there are atoms in the visible universe.
reply
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.
A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year [2], according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.
[1] https://www.sciencedirect.com/science/article/pii/S089662732...
[2] Assuming 16 waking hours, and 10^9 bits/s of input that the paper mentions. 16*3600*365*10^9 = 21.024*10^15
Something that more or less guarantees you’ll get about the same capabilities as the original model, but with a different final expression of certain characteristics
I wonder if this could be a good distribution model too, selling “dried” models (almost like raisins - or the beings from the 3 body problem), and then you can buy them to fully train (raise?) with your own data
It is not. There are many biological examples of random processes generating patterns and weights, such tiger stripes, fingerprints, etc. DNA only encodes the reaction, but the pattern itself is seeded by initial position of random molecules, not the DNA.
> that it influences the initial state of the brain.
You've gone from "DNA is probably the seed for the random initialization", as in the deterministic seed for a PRNG, to "influences". Those are 2 different claims.
Most scientists will agree that DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc.
Most scientists will disagree that it is "probable" that DNA encodes/seeds the weights of each and every synapse.