I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier concepts like compressed latent attention. I can follow the field and understand it.
Not sure I would have got as much out of it at seventeen. I have a lot of background and experience which made it much easier to learn. I wasn't struggling with the linear algebra or with python. I already knew pytorch and neural networks. That helped a lot and I covered these tutorials fast and could skip over large sections. A few evenings and the odd weekend day over a couple of months was enough for me.
For seventeen year olds the tutorials are good enough to make it possible to learn this but it would have taken a lot longer to understand. On the other hand I would have learned a lot more. I think I would have learned a lot of valuable stuff.
However I also think 17 year old me was studying for his A levels and probably this was right choice in terms of maximising future opportunities. I'm not sure I think learning about LLMs instead is sensible. Indeed it might be bad advice. But I can absolutely agree with the sentiment.I think 17 year old me would have wanted to do this too.
Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?
Personally I don't see the problem, as long as you're aware there is survivorship bias involved here.
What's the alternative really, seek advice from unsuccessful people? That seems worse :)
Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do whatever the fuck I want.
Agreed, my previous stated "ignore both and do whatever the fuck I want" approach has worked out very well for me in life, people should probably focus on identifying better what their gut tells them, rather than what randoms on the internet thinks and writes.
Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.
Nonetheless, there are many successful people I would gladly listen to for advice, though they are often successful in a different meaning than what venture capitalists would use (e.g. parents with great kids, managing to keep a healthy work-life balance, happiness, and maybe even having time to spend on some cool hobby project -- you are heros!)
Not sure why would you think so.
Inverse reasoning is very powerful, and unsuccessful people can tell you teach you plenty of "don't do this mistake", which the survivors would not even think about.
But how can I know for sure that that particular mistake is actually why they were unsuccessful? Has exactly the same issue as listening only to successful people as they hardly know what actually made them successful most of the time, but they still compose large blog posts with their reasoning for why.
Again, I still think my approach of reading both but then regardless go my own way is the preferable approach, at least for me, ymmv.
That’s what you get from listening to “successful people”. You get to learn about all the things they tried that failed, then the things that did work on that 24th try, which was successful.
The “survivorship bias” people always seem to assume that the “survivor” lucked into his fortune on his first try ever, so he can’t have learned anything, so we don’t have to listen to him. But that’s seldom the case.
I’ve written about this before:
https://www.youtube.com/watch?v=3LopI4YeC4I
An advantage that is not "advisable", like being born in january, in a rich country, in an above average family, or just having luck, might have more influence on the outcome than any conscious action. It is almost sure that one-in-a-million level people only edge over the other 999,999 they competed with is just "have more luck".
how many of us out here are doing work directly in what we got a degree in? I majored in economics and now I'm a CTO.
I would absolutely advise a 17 yo to learn how to code, understand how transitors work and how to code an llm. even if he never works on llms, you basically end up with a kid with applied knowlege of statistics, math, physics hardware, logic and a whole lot of practice in critical thinking.
I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.
Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.
But of course, 10 years ago this wasn't obvious.
I’m no paulg, but if you’re reading this - and you’re 17 - just focus on getting into a university and having a good time that you won’t regret later. Play games/sports, make relationships, fall in love, explore.
Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.
Hazarding a guess, I think the moment Altman became the CEO and later during COVID, the sentiment seemed to have been shifting towards what we see today. But this is all based on hazy memory, rather than looking at the data. I'm sure there is a blog post waiting to be written about analyzing the sentiment of comments to PGs articles on HN, and you'll see a shift somewhere.
Hard disagree. This submission is still being highly upvoted, while another recent post[1] on the harms caused by Graham’s fellows[2], with a fairly tame comment section, has been flagged. That is a constant on HN. It’s not a fluke, it’s as predictable as the sunrise and getting more pronounced.
I’m sure we’re both biased in our perceptions, but as someone who only learned of Graham later on, my perception is that HN in general (certainly more than any other website) used to worship[3] everything he wrote, together with others like Musk, until things started to really go to shit and many eyes have been opened to the effects of the unfettered greed of rich tech guys out of touch with reality.[4]
[1]: https://news.ycombinator.com/item?id=49411762
[2]: A better English word to describe what I mean is escaping me
[3]: That word I choose hyperbolically but deliberately. It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.
[4]: That is not “knee-jerk” but a slow realisation still ongoing.
No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.
But why are people cynical? they ask.
Don't many of the commercial ones prevent you from using them to build LLMs?
I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.
http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.
There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.
2. LLM from 0 to Hero, and nanoGPT by Andrej Karpathy
(Edit: And learn how honest business works)
I think the last one was seeing a skilled electronics repairman do surgery on a CT machine controller.
Might be that these people are from outside the US as well, where things like "honest business" is very much possible today, probably most businesses I interact with AFK on a daily business are "honest businesses".
I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.
The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.
Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.
17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.
If the two current bottlenecks, for this LLM madness that could very well be a bubble, are processing capacity and accuracy (a second processing problem) then what comes next? Isn’t that where young people should be looking or are we just giving up on innovation?
I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:
If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.
Wait no it’s not, that was always happening.
What’s crazy is that people still believe in it.
And it's basically a weekend project to put transformers together in a ML library and train it.
The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?
Even worse when they ask themselves.
Either way, this isn't really advice for 17 year olds. Pg is thinking out loud about the pathways for founders.
>Whoa. I’m 19 and I trained a 100M language model from scratch. Did a v2 now with a new SFT experiment to see if I can get better results on same size.
Owner of Golf Club Company says I should dedicate my life to golf lmfao.
I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.
If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this.
Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is or why they want to explore that, just let people explore, regardless of age.
It was such a terrible experience being a young kid growing up, with so many adults spending hours trying to convince me to stop sitting in front of the computer so much doing whatever; "why are you even trying to learn that stuff, you have to go to school to understand anything of this" and so much other similar trash.
Sorry, not your fault and I'm borderline trauma-dumping now, but really sad to see this sort of gatekeeping on HN of all places, age is irrelevant to learning ANYTHING, in my humble opinion at least.
Kids, find anything interesting? Jump into it, ignore what adults tell you, and do whatever you feel like, you'll find your place eventually.
This also meant that by the time I was actually offered to take a programming class in school (junior year of HS), I had already been able to self-teach myself well beyond what that class was covering, thanks to just working on random projects that scratched an itch I had at the time, looking up anything I didn't know or understand, and internalizing those concepts over time.
In short though, I definitely agree, young kids and teens (and also, frankly, adults too!) should be encouraged to explore things that they have a passion for, without being told 'you need to go to school for this' or 'you cant understand this at your age'
There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.
Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.
> youngsters who'd be able to extract tremensous value from such an excercise
But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.
This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
To you it isn't, is my entire point here. But why extrapolate what you think is fun, to others? Sure, I don't find that fun either (although useful), but who am I to say it isn't fun for others?
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
That's what I would call exploring.
I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
It is not how LLMs are "built from scratch".
I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.
Transformers are difficult to understand even to people with strong ML background, let alone a teenager.
That 17yo would have already built many uncommon bases, and would build further.
That is, a 17yo with proper mentality.
Can you provide an example?
The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.
Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.
With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.
Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to more resources. It has very good prospects if you do manage to push through. But it's a hard and risky path, and you will not be able to get any interesting results for the longest time.
For a young middle-class student, it just doesn't make much sense. You can do much more impressive and impactful things with your time without getting into that black hole.
I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. But they can't do much with it without capital or access.
Good engineering has never been a bottleneck in this field, it's been all about having access to capital and taking smart but dangerous risks burning it on compute, without much idea of how long you need to keep burning for. There's still no end in sight, some are still managing to convince investors and keep burning, and we are seeing progress, but the business case is still unclear. If you want to get in that game, go ahead, but it's not something I would advice the average young engineer.
The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters.
Due to how much high quality data is readily available, anyone can now train a sub-billion (L?)LM on commodity hardware.
And I'm personally convinced that pretty much any enterprise use-case of an LLM (except coding) is better served by a fine-tuned small (<2B) model that is trained specifically on the task, rather than a generalist frontier model, so learning the engineering around fine-tuning is a key skill that companies will realize they need sooner than later.
I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes.
On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
The world needs pragmatic folks who work at a higher abstraction and make LLMs useful, AND also folks who think why not “this other way”? And build newer ways to do fundamental things.
Given the usefulness of current LLMs, I would certainly encourage anybody to try and build their own LLMs, and see what they come up with…
Heck if they build a rack full of old laptops and run something with it that could be done “better” with modern servers, I’d still appreciate the learning running things on those little machines bring.
You can however learn everything you need to know to get on the career ladder as a software engineer on a regular home PC.
Do you take the first step or rule it out because you don’t yet see the complete picture.
As a teenager I never hesitated to try things out. As a young adult I wanted the whole picture. Now I’m back to playing / trying things out. I kinda wish I’d not given it up. PG being a bit older and reminiscing - I bet he’s in that bucket too, whereas someone trying to establish themselves professionally probably (aka me early 20s) wants to see the path.
The only jobs that he found he was highly over qualified and they paid very little.
In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.
Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more work, some of them even trying to recruit me to work for them full-time as well. I'm based in continental Europe, fwiw.
Also bunch of past workplaces who've adopted AI in various ways who reach out once they find out what my current focus lies, but that's harder for others to replicate unless you've already had a career as a developer.
I don't think the "incredibly small minority of companies in the world do any real training or optimisation" part is necessarily as true, as some parts of the work I do get is about helping them optimize training and infrastructure around training. Mind you, none of this is for building LLMs from scratch, it's 99% fine-tuning existing checkpoints.
I'd also agree with "paulg is in somewhat of a bubble" regardless of this, which is worth remembering whenever you read his content. Same goes for any person living in SF, and dare I say the US. But also, YMMV, I live and work in Europe, probably why I have this perspective.
You’re missing the point. Understanding how Unity works fundamentally makes you a better Unity dev.
1. Both training and optimisation will get significantly cheaper and easier quickly.
2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029.
3. The big AI firms will become part of the surveillance capitalism network, if they're not already.
So I think for self-protection a lot of companies will be looking near to medium term AI independence.
For the time being, unless you truly have millions, the outcome from training will be very net negative, while focusing on building on top of existing AI will yield amazing things if you apply the same talent and effort.
When it does get cheaper, then it will be easier to acquire the skills and experience too, and the struggle you went through by trying to do it now will be somewhat wasted.
Besides, I am well versed in this field, and it is not rocket science. There are plenty of software engineering domains that are a lot more challenging, like high-end graphics, large-scale data engineering or kernel programming. People will learn to train LLMs when people want them to.
In reality, enterprises are happy to offload even risky tasks to others as long as they get some contractual guarantees about their data. Would they like more choice in who to buy from? Yes, but not enough to in-house such a specific discipline.
Fine-tuning a model or LoRA based on the companies data set is more feasible but you're likely going to need several runs as you test/try out different base models, parameters, etc. This is why there are a lot of fine-tuned models on huggingface based on base or instruction-trained models from the larger AI companies that have released open weight models (Microsoft, Google, IBM, Mistral, DeepSeek, Qwen, etc.).
Training is limited on memory first (storing training data and weights) and computation second. Realistically you need to own or rent 2-8 H100/B100 devices or Google's TPUs.
The majority of workflows for a company providing AI capabilities are likely best solved by tailoring a system prompt for the chosen model, evaluating the prompt and model with tools like promptfoo, and then running it on a compute cloud provider (including AWS Bedrock). If the company is big/financially well off enough they could look at buying the hardware needed to run it on their own servers.
For other uses like agentic software development you'd need to spin up a suitable model on a compute cloud provider (or local hardware if the model is small enough) and then tell your IDE/editor to use that model. You would need some way of benchmarking and evaluating the models to see if they are capable of doing the tasks you need. -- There have been some tests done by people on YouTube that suggests that Qwen 3.8 27B is a decent model, but your needs may vary.
There are plenty of areas were we need people to do this for insurances, banks etc.
AI/ML exists on many levels.
In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% better architecture, it just doesn't matter. (and never mind that even OpenAI hasn't really solved a voice model yet. Try it. It can probably match 2026-quality call centers, but it's no substitute for an actually empowered human)
... and yet, if you look at what hyperscalers are getting paid for ... comfortably more than half the income is training. Which makes no sense on so many levels.
e.g. https://valueaddvc.com/blog/inference-chips-vs-training-chip... (I get it, not great first source, but st
For some very niche cases I think this is probably the case but for the vast majority, the company's data isn't as useful as they think it is or anywhere near the size needed.
I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if you try to train, you cannot get anything amazing unless you burn millions.
Very few people can afford to play the long game and cross that dessert. And, sure, you will not get far without good engineering, but good engineering is definitely not sufficient and is not the primary bottleneck.