The Death of Websites: David Siegel on AI's New Economy

Hey dad, you know the first AI class I ever took, and this is true, is 1983 at Stanford. Stanford was doing AI in the 80s. The graduate AI class that I took and didn't understand any of it. What did they do in the 80s? What are they diving into in terms of AI? I was a computer science graduate student and we had the Stanford Artificial Intelligence Lab. This is kind of the beginning of machine learning. The famous example they used is that the first day of the first class, the professor got up and wrote on the board 'how to wreck a nice beach' and then he explained that that was how to recognize speech. Interesting. And it was hard and the speech recognizers in the early 80s got a lot of things wrong. The question was how are we going to make it better. That's kind of what AI was at that time.

There you go. That's not large language models, that's machine learning, and we've come a long way but only because of the persistent step-by-step work of people since the 70s, since the checkers program which was written at Stanford and the early chess programs. All of that was just building on top of that. One thing I want to explain is that today's large language models are built on transformers and all transformers do is predict the next word. Most people have heard this, they don't know exactly what that means. Google has been completing people's sentences for more than 10 years in Gmail, predicting the next word or a couple of words. That's very serious software they've been working on since before transformers, how to predict the next word. So it's no surprise that the big paper in 2018 came out of Google on what the trick is. The trick is attention. If people don't know about attention they can learn about it in my video series. That is the thing that let them predict the next word. The other power besides Google that was also predicting the next word was China. Because Chinese has like 60,000 or 80,000 characters. They've been able to type it out on a QWERTY keyboard for the last 40 years. How do you get Chinese characters out of a QWERTY keyboard? They type the sound of the word. You type out the English version of how it sounds of the character. Like 'mei' means American. So you type 'mei' and then it puts up a little map of characters that start with that and the thing that it's most likely to be in your context of what you're typing and it highlights that word for you. All you have to do is hit return and it puts that Chinese symbol in. It's all about in-context prediction. That's why Google and the Chinese government are the two at the front of this because they've been predicting the next word for 40 years already. For Google's case, for 15 years or something.

They are, that's the foundation of that. That is why Nvidia is the highest valued company in the world. Because the way a transformer figures out attention is it has to load the whole sentence into memory at once and process everything in parallel. So you have to do parallel processing which means you have to have a GPU. Nvidia was just making money mostly on Bitcoin mining chips at the time. Then boom, you have to blow the whole thing in and process everything in parallel. So their demand went as soon as that paper came out in 2018 called 'Attention is All You Need' from Google, the floodgates opened and Nvidia scrambled to keep up with demand and still is. Competitors to Nvidia are starting to catch up. AMD is building out data centers just as fast. Intel, what do they do? They build single-core CPUs. But they've had to get fast, real smart, real fast to build GPUs because that's what it takes to process attention and that's what we're going to need a lot more of. This is all just by way of some interesting technical stuff. I do want to talk about hallucinations because people know there are silly hallucinations, but we don't know how it happens.

They are. I can give some hints about this. It's in my video series, but I'll give some hints. First, how do they happen? It can only guess the next word, one word at a time. If you said, 'How do Americans like their coffee?' It will say 'Americans like their coffee' and then the next word is either 'black' or 'with'. Those are the only two. If you look at everything anybody's ever written about how anybody drinks coffee, the two big candidates are 'black' and 'with'. It has to decide between those two. It does it by the weights that it has of all the things it's ever read, consumed, ingested, processed about coffee. Most likely, there are a lot more people who like it with sugar or milk or cream than black. So the most likely word is 'with', but 'black' is a close second. There is a parameter that I think everybody can really learn a lot from. There's a parameter called temperature. That parameter goes from 0 to 2. One is where it's normally set. Zero means when you get to the choice of the next word, you must take the statistically most likely next word. If you ask it the same question 20 times, it'll give you the exact same answer 20 times. If you set it to 2, it's got a whole range of potential next words. It'll say 'Americans like their coffee Fox' or 'Bowling' or whatever anybody ever wrote in some sitcom. That's why we normally set it to one, because one means one standard deviation. Within the top four or five or six, depending on the distribution of what the possible words are. By setting this temp variable, you can control the hallucinations. When you want to be more precise, set it low, like 0.2 to 0.4, and you'll get more logical, less wacky. As soon as it says 'Americans like their coffee with', now it's ruled out 'black' and it's on a new adventure. It's got to get its next word. After four, five, six words, if the temperature is pretty high, you can get pretty far afield from where you were trying to start. That is how they hallucinate pretty wildly when temp is high and they will hallucinate much less when temp is low.

I'll tell you. From the weights that it gets from all its training. It reads billions of books and articles and blogs and National Enquirer nonsense. If you ask about blue zebras, somebody somewhere has written something about blue zebras and there will be a relationship between blue and zebras in the training data in the weights. It'll give you whatever a couple of people in a tiny number of documents in the world talk about blue zebras. It's not a dream, it's not made up, it's not random. It's coming from whatever it's been trained on. If you ask about airline timetables, that's the training data. If you ask about exotic French menu items, it's going to come from that training data. If you say 'Tell me about Moby', it's very likely to say 'Dick' because there's a lot more written online about Moby Dick. It'll assume Moby Dick was a novel written by Herman Melville. It comes from the overwhelming weight of training data.

Let's do a thought experiment. Suppose you have a favorite non-fiction author. Maybe dead or alive. Do you have one?

Carl Sagan. You get Carl Sagan on the phone. He's alive. You have a fascinating conversation with him. It's inspiring. He's answering your questions one-on-one. You can ask him the deepest darkest secrets. That's a big value add for you to be able to talk with him personally. Now take the same thing and have that same conversation with your favorite author by email or chat. You're learning a lot. You're asking specific questions about your situation. You're getting all this nitty-gritty stuff that you can't really get from a book because it's combining different perspectives. After 45 minutes of that, you're like, 'Wow, that was amazing.' What is the difference between that stream of words going back and forth and the stream of words that comes from a prompted LLM pretending to be that person or simulating drawing from that person's whole body of work?

Today I had a conversation with Grok about the chance that AI will kill everybody in the next 50 years. I asked it, 'What do you think is the chance of AI killing all humans within 50 years?' It broke it down and said realistically there's a 10 to 20% chance. How did it come up with 10 to 20%? Other people say 50, 70, 80, 100%. People like Sam Altman say 10%. How does it come up with 20%? By consuming a lot of training data of what a lot of people have said. It's amalgamating all that data from a Bayesian perspective by weighting everything and putting all those opinions onto a scale and it comes out around 20%. That's Grok's opinion. You could ask ChatGPT the same thing and it might come up with 15% or 12%. It may have been trained differently. An LLM is trained about 75-80% on the data out there and about 20-25% with humans trying to correct it and add their own. This is kind of where they get in trouble if you ask me. They're all either in California liberals or they're in India hired by California liberals. Different LLM companies will have a different approach. If you really want hardcore logic and thinking, I would go with Claude. I prefer Claude when I want really rigorous. But it's still just predicting the next word. Although it sounds a lot more like thought in the same way that if you emailed Carl Sagan and got an answer, it would sound a lot like thought. I always say at the end of a discussion with an LLM, 'What do you think?' The answer's pretty interesting and getting better. 18 months ago, when you would ask ChatGPT or Claude for a thing, it would give you its answer and that was it. Now it's saying, 'The next logical step would be you might want these three things. Do you want me to do any of those things?' More agentic, more forward-thinking. One tactic that's happening is they'll take multiple passes. That's what thinking mode or deep research is. It'll take a first pass and then it'll look at that and see if it has any hallucinations, see if it can get anything better. I have this in a system prompt. It says use red team thinking, come up with an answer and then find ways that it could be wrong. Google has just written a paper on this in Science. They have a multimodal approach where they'll take a hypothesis and test it in many different ways. They'll analyze the data or the result using half a dozen different ways and then they'll get half a dozen different answers. Then it will synthesize a sort of Bayesian weighting of all those different answers and give you a weighted response. You can also say, 'What would Milton Friedman say?' I heard somebody tweet something today about companies firing thousands of people while the CEO still got paid a lot of money. I gave it to Grok and said, 'What would Milton Friedman say?' It said Milton Friedman would say that the CEOs are adding value because by not firing those people, they could put their company at tremendous risk. There are dozens of examples where the CEO was too slow to fire. If you ask that same question using somebody else's perspective, you'd get a different perspective.

There's two answers. First of all, how smart are you? My favorite living thinker right now is Brian Caplan. I read everything I get from him. I'm in touch with him. He's a thousand times smarter than I am. I'll take his opinion over mine every day. On the other hand, I like to develop my opinion. I want to blend. I want to get that and see, do I agree? I really like choosing people who are smarter than I am to get answers from and learn from. Second, there are ways to add your information. This is actually helpful for business owners if they want to have themselves on their website. There's a company called Delphi, deli.io, that does this. You can upload all of your everything you've ever written, all your blog posts or even recorded speeches or phone calls. You could get transcripts of everything you've ever said. It's all your data. If you've written books, if you've written ad copy, whatever you want, you can upload it and then it will RAG that. RAG means it will stick it on top of an LLM and take those answers first. It will prioritize your information. If you're good at sales and you've got a website and you want to talk, or you're good at gardening or construction, you can have your own little talking head talking to people giving your perspective. Then it will fill in stuff you haven't given it with other things from the rest of the web. This is what our program does at Redshift Labs. We can take that and turn it into a sales bot. It can be your best salesperson. It can sit on your website and give your thoughts, your content, and your sales approach right to your customers. That's one way, this RAG concept which puts your information first.

I wrote a book about what's next in 2010. It's this book right here. It says 'Semantic Web'. Nobody knows what the semantic web is because it didn't happen because Google killed it. There were a couple hundred of us trying to make it happen. The semantic web was based on really understanding the meaning of words in a machine sense so that machines could make sense of all the crap that we put up out there. It didn't work. Google came along and used their technology to sort of AI machine learning make sense of the world as it was. This is called unstructured data. I wrote this book to try to get people to understand how different the world would be when we actually understood what we meant. A good example is a truck moving from Italy to Sweden goes across six borders. Every one of those borders has forms to fill out and rules and regs. If it's in English and with forms on paper, it adds up to $10,000 worth of overhead cost just to get a truck 300 miles across Europe. If it were all semantic, it would just flow. The systems would just give a green light, you just move through. No red tape. For example, if you're trucking wine across the country, you can have sensors on every bottle. If the temperature goes up a certain amount, that would be reported to the buyer. You wouldn't take any chances anymore. The reason the book is called 'Pull' is that if you made everything smart, you would change all businesses from pushing the supply chain to pulling. I used the example of music. Music was dominated by the record labels. They would work with bands for months to put out an album and then the album would come to Tower Records. That was the supply chain. Music switched first and it just became pull. It became on demand in your little device. It dematerialized. When that happened, it switched the delivery mode from pushing to pulling. I wrote this book saying industry after industry, wake up, do this, get ahead. It didn't happen, but now you're seeing it start to happen more. Now we have a full reset. That's why my video series is called 'Reset'. I think it's now time for most businesses to go this way. You should be able to, if you like a movie like 'The Fugitive' with Harrison Ford, but I don't want Harrison Ford. I want Billy Crystal or Brad Pitt. Make that movie with all these different people and I want it on my screen in 10 minutes. It'll make the next frame one frame at a time and give you that movie on demand. We've already got AI music that just plays sounds and notes that are pleasing to you. All this music is original all the time. It's just coming to you because that's what you like.