I read it all. There are no shockers in the boxes. It's all explained ahead of time and by the time the contents of the boxes are revealed, you'll wish you didn't read all of that.
I interpreted it another way: the boxes are what connects the past to this written story. Without the boxes most details would be forgotten and this article wouldn't exist. The documents in the boxes were only thoroughly read by one daughter starting in 2024.
Agreed, he’s saying if he’s working with a black box he doesn’t understand what it’s doing. Which is of course true. If the black box were a chinese person instead of a tool and he was writing down what they said, would he say the chinese person in the black box does not understand why they told him to write something? No, of course not.
Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.
- A motor is something that create a force to push a vehicle.
- Oh yeah? My neighbour car does not have wheels and sit on concrete blocks, the vehicle does not move and yet we all agree it has a motor. So it means that I can claim that this other thing that does not move has a motor too.
Sure, human can _some times_ not do some stuffs, but the fact that they can do these stuffs sometimes is the point.
Doing these stuffs is the hard thing. Doing these stuffs is the proof that the machine has what it takes. It does not matter if someone cannot do that stuff, it does not imply that their internal system is not complex enough to potentially do it. But the fact that some people can do that stuff is the demonstration that inside a human skull, there is a system that is complex enough to potentially do it. Unless you can prove that people who don't do it have a fundamentally different system inside their skull, then you cannot pretend that they should be considered as having a less complex system.
Human _can_ check themselves. They don't _always_ check themselves.
Motor _can_ move vehicle. They don't _always_ move vehicle.
LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
So, yes, it is a refutation. If you have something that _never_ can move a vehicle, this thing does not qualify as a motor, even if some motor, sometimes, don't move a vehicle.
And if your next argument is "yeah but I would argue you don't need to check yourself to be conscious or to understand things", then you just redefine the definition that is owned by your interlocutor. Your interlocutor is saying that this is a criteria they are expecting. Good for you if you are not expecting this criteria. But the problem is that the answer is not "this criteria is not expected", the answer is "I change the criteria from 'being capable to in some circumstances' into 'does always do it in any circumstances'".
> LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
All modern agentic harnesses can do this. Nobody uses raw LLM for anything remotely complex. There's always some external system in place. That system is part of the "thought process".
Adjacency doesn't matter here, only what the result of the system of pieces is.
It means having self-control on their action and being aware of them. If you ask a system, it will respond, it cannot choose to not respond (even if the response if "I don't want to response", it still "run", still do the work). If you don't ask a system, it will not respond.
Adjacency is the point of the thread here. Saying "you say X is important to decide if the thing is intelligent/understanding/conscious, so let me just change X in the middle of the discussion and say that X does not matter".
That is exactly my first comment in this thread: I don't care if AI think or whatever, my reaction was about these "counter-arguments" that totally miss the point and make the person who push them ridiculous. If you want to have a counter-argument, you first need to understand the interlocutor, not just spew whatever rebuttal you constructed that answer something unrelated to what the interlocutor brought to the conversation.
In my thought process, I quite literally stop myself, and say "ok, think about what you just said" to check myself. I literally initiate that loop. If I don't, then I'm not using my own mental agency, and just using my firm coded priors.
I will say that I do seem to have a stop, what you said is wrong logic check voice that pops up without me initiating it. But, it's unreliable, and not too much different than all the content monitoring system used for the streaming clients, that will terminate with "content violation" immediately after the "incorrect" words are sent. I don't think integration is important, just the behavior of the overall system.
There is no "loop" in the brain, it is all part of a same line of thought. This is visible because, while you can sometimes have a "two voices / dialectic" way of thinking, you can have the exact same thinking in one-go that does not look like a loop at all.
In fact, in the large majority of the time, you don't process "as a loop" at all, you just continuously progress in your reflection without needing a "second voice" to retrigger you. The fact that sometimes we do this is just something we can do, not the result of something needed for our brain to work. For AI systems, this is something needed because the "answering" part is not able to do the loop on its own. And building a bigger system that combine an "answering" part and a "loop" part does not fit this, does not create a self-reflective system, it just makes a non-self-reflective system and a workaround bundled together.
It's a bit like if the "answering" part was unable to provide only one answer and was always producing plenty of different possible answers, including contradictory ones. Then, you can add an external part that will just pick one answer (and add it to the context so the next large set of answer will not be inconsistent), without any intelligence to it. The whole system will look like a human. But we know that the system is not "living" and "aware", because a "living" or "aware" system has its own opinion, while this system is just generating convincing sentence without seeing any hierarchy or value or meaning in each one.
I would claim that, if you think without introspection (that loop), then there is virtually no self check. I'm not sure what "self check" you see that the brain has. Could you describe this "self check in a line of thought"? How do you perceive the check there? This is a genuine question. It definitely doesn't align with how I think about things. I ponder and talk to myself to iterate verify and test my understanding of my own thoughts.
Maybe a good analogy is "throwing a paper plane in real life" and "throwing a paper plane in a video game".
In real life, the paper moves "by itself". It does not need an external loop that update its position in a loop manner.
In the video game, you need an internal loop, a step-by-step tick, that update the plane position based on its current position and its momentum. And this is why a video game paper plane is not a real object. It is a very good simulation, it looks like it, but it is missing some intrinsic properties that we expect from a real object.
Yet you can analyse the paper plane trajectory and see it as a Markov chain, with quantified step-by-step progress (for example one position point every 0.1 second). The same way you can look at your though process and identify a step-by-step progression. But it does not mean that it works like that intrinsically, it does not mean that the paper plane "jumps" from position point at time T1 to position point at time T1+0.1 second.
For the human brain, there is no "loop centre" in the brain. There is no one (to my knowledge) who got a brain injury and suddenly were unable to keep a single line of thought without having someone else having to feed them the previous thought in order to feed the next thought.
In the brain, the fact that the previous thought feeds the next thought is "how it works", it is intrinsic, it is by design. And this mechanism of thoughts feeding the next thoughts is what creates "consciousness" or "awareness": self-reflection is based on the fact that thoughts are intrinsically linked together, that they "flow" continuously, without needing an external system to update them.
You cannot take away the "loop" part of the paper plane so that it suddenly would be unable to move on its own once thrown away.
Now, you can always say "well, the paper plane in the video game is a very good simulation, it does not matter if it is a real object or not", and that is fair enough. But in this discussion, some people have arguments to support that this property matters, that it is one condition for consciousness or awareness.
Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?
I think the result of the system is all that's important. Where/how it's implemented doesn't matter for practical results.
If the argument here is that LLM don't have this built in, you should know that nobody has a practical use for plain LLMs these days. Nobody uses them this way, except for debug. All interesting use is through some kind of harness, with all sorts of systems bolted on. I think these conversations are only meaningful in this "agent" context that people actually use LLM, where they stop when they think they're done.
LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.
Have you read the article in question. It is saying that for one continuous thought, the brain will use different part of the brain to do different thing. It does not say that there is a "loop controler" anywhere. On the contrary, it illustrates that there is no loop controller: there is not special brain function that control this loop, this loop is "how the brain works", and LLM don't do that, they are incapable to do that, it is not how they work.
> Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?
No, my argument is that the nature of the brain and the nature of the LLM are very different, as different as a real paper plane and a video game paper plane. Some characteristics (for example, awareness) that exist in the brain cannot exist in the LLM because these characteristics are the result of the nature of the thing in question.
The problem is not that you build a system by integrating 2 things together. The problem is that they are different "things", they are different machines, they function, fundamentally, differently. They may produce the same output, but when you say "the brain has the characteristic X, the LLM produce the same output, so the LLM also has the characteristic X", it is logically inconsistent.
Planes are built as a system combining 2 things: a motor and some wings. But they are fundamentally different from a bird. They just don't "work" the same. It is not the same mechanism.
> you should know that nobody has a practical use for plain LLMs these days
That is totally irrelevant. My point is about the nature of the LLM, and the fact that it is stupid to see the same output and to conclude that they have the same characteristic. It is like saying "Birds are flying in the air and are alive. Planes are flying in the air, so I guess they are alive".
> LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.
No, you miss the point. The problem is not that "you can just add an external loop". The problem is that the brain is a system that works without such control loop. The thoughts are flowing (and they may flow to different brain functions, like explained in the article you quote). It is part of how the system works. Having a system that contains 2 things, one that does one computation and one that control the loop is not equivalent to another system where you cannot decouple the "flowing of the thought" from the "thinking machine".
Yes, LLMs don't think on their own, for one; they think when you invoke them.
My rebuttal is that people only think when invoked just the same and can enter states where there is no consciousness just the same. The OP has already accepted that LLMs think, but it seems that you are arguing they do not? This car business is confusing and the LLMs not checking themselves is also wrong, there’s even a benchmark for this
https://correctbench.github.io/
What you said: I have example where, sometimes, human think when invoked.
That's the difference: human brains are intrinsically different because they are built to be able to think without being invoked, even if there are situations where they think when invoked.
There are tons of obvious examples of human thinking without being invoked. Just take a bath and you will see :)
To be clear: the person I was replying to asked if the way a human thinks was any different from an LLM with a context window. That's the context of my answer. An LLM is a machine, it can't do anything unless we invoke it or give it the instructions and capabilities to do so. It has no free will, it can't just decide to compose a symphony one day unless those are part of its instructions. It can't do anything unless we tell it to do so and give it the capabilities to do so, it doesn't even exist unless it's loaded into memory. That's obviously different from human consciousness, and that's the whole of the point that I'm making.
You can argue that humans are just biological machines reacting to external stimuli, but that's a philosophical argument that I'm not interested in having and frankly, I think it'd be selling yourself short a little bit.
Thank you I appreciate your opinion. I do think that we are reacting to external stimuli even though our ego is uncomfortable with being deterministic in any way, which is the free will point that you hit on. I think that is likely to be the point that keeps the argument going as it’s not a settled debate absent any AI, which we clearly all see as either deterministic
or semi-random when the temperature gets turned up.
As far as the argument about being loaded in memory, if there’s any consciousness in AI, it’s obviously in different form than a biological consciousness. We’d have to agree that consciousness does not require a body to get past this.
AI does not react to endorphins and other hormones so we know that our minds and bodies are influenced by other forces and in other manners. An AI won’t be frightened or angry or aroused, but when those things happen to us they are usually called ‘subconscious’ reactions.
Should we require the subconscious to be part of consciousness?
On top of that, neuroscientists find that your mind backfills the conscious reasoning for your reactions after they happen. This is known as our consciousness being the left brain ‘interpreter’ https://en.wikipedia.org/wiki/Left-brain_interpreter
Reading that, an LLM can serve as a ‘left brain interpreter’ as that’s exactly what they are designed to do - come up with reasonable explanations and fill in the blanks based in the data provided.
Right, the export controls are only forcing Chinese AI to innovate, build their own fabs, and make training and inference more efficient. The end game of this will be NVIDIA chips won’t be wanted because you can get a $50 chinese chip running a ternary model that is competitive with claude in English and is much better in Mandarin.
The US government has failed to learn from its own history.
60 years ago the US government had forbidden the export of fast computers to France, with the hope that this sanction will prevent the French from developing thermonuclear bombs.
The result was that the French state (which at that time was lead by de Gaulle, not much less autocratically than China) subsidized some of their computer manufacturers, which previously could not compete with the American companies like IBM and CDC, and also their semiconductor manufacturing industry, which had to provide the components for the locally-made computers.
Eventually, the French produced TTL circuits and mainframe computers made with them, and finally they also made thermonuclear bombs.
So the American "sanctions" against France have been a complete failure and have been great for the French industry of semiconductors and computers.
Many years later, when USA no longer had export restrictions towards France and the French state no longer protected their industry, the French industries of integrated circuits and computers have been greatly reduced, their companies either becoming bankrupt or being bought or merged into multinational companies.
When De Gaulle did ask the french via a poll if they wanted him to leave, they said yes, and he left. He is also the guy who did setup the balance between the various political powers, which has been kind of working... until now (currently the government can hardly get laws from the parliament, because few people representatives are on gov side, and they won't die or disappear if they disagree, bugger!). The fact that the president must leave after 10 years is kind of recent though.
France has always been a very strong US ally, in an honest relationship, namely without agreeing or being on board with everything.
And France never had the intention to nuke the US... unlike some other country we talk about all the time in news (that said, France is not far from the US on their list...).
And compared to the rest of the world, don't forget the 'western world' (which is not 'western' only anymore...) has very, very close core values. A good way to think about it: a big dysfunctional family.
On the software side, aka the 'silicon master control' side: currently, the french are just Big Tech slaves. To be more current, Holland president and Valls prime minister did install a document (2015/2016) which has been "law" since which literaly "pushes" (hard) administration online services to be hardcore dependent on Big Tech (mostly the whatng cartel) without any reasonable technical way out (unless noscript/basic HTML web sites, are brought back in the security infrastructure, like they were a few years back). This document is out of reach of even the parliament, namely only the president and prime minister have control over it, in other words, to interact with this document you need the same level of power required to decide to increase the number of atomic bombs(huh). The following president and prime ministers did nothing and kept increasing french administration dependency, I guess they were/are as guilty OR BRAIN WASHED than Hollande and Valls.
Open source does not matter anymore (look at how big tech controls open source software via often-non-pertinent complexity and size), _LEAN_ open source does, and that includes the SDK (aka the computer languages: if you need a giga huge and complex compiler, you already lost).
On the hardware side, state-of-the-art chip is an international effort with an insane supply chain. This is mostly 'driven'/hogged by US chip designers. State of the art, foundries are currently in TW (the US is working at getting some back), EUV is from EU-ish (the EUV light is from the US), and many, many more high-hech tools are from the US/JP/TW/etc.
What I am wondering: did Holland and Valls "give" France to Big Tech... or "sold" it, if you see what I mean, because it is very easy to setup public money channels using 'Big Tech' which look "clean", aka hidden behind a technologi-blablublo smoke screen, since most people are scared of tech and/or don't understand the fine details.
It is all about simple file formats and network protocols, good enough to do the job and stable in time. A good compromise is to use a strongely and dynamically defined subset of Big Tech stuff, which you know can be locally implemented with reasonable effort (by citizens, small companies, state administrations, etc). That will foster alternatives (good I guess). That's why I am talking about web sites, and not web apps (noscript/basic HTML), and we could talk about a strongely defined subset of PDF.
Ofc, the devil hides in the details, this is a very coarse overview: you have to basically decide in a fine-grained case by case, mistakes will be made and will have to painfully be fixed. You cannot get it all in one shot, it is module per module, back and forth, and probably slowly.
I would order that in a heartbeat. Even if it required proprietary Chinese-government drivers. I would try to segregate in a VM without internet or something. Please make this happen! Tokens cost too much in the current system.
We have GPU costs, power costs, and how many token/s models can generate on those GPUs. It’s possible to figure out the marginal cost based on this. The current estimate is about $0.40 per million tokens for gpt4 equivalent model. Sonnet 4 is $15 per million tokens, so they are charging high margins on inference. The issue is how large of a margin is needed to recover their costs before the GPUs age out, and how high of a margin can be charged before it’s not economically viable.
I skimmed the article, but couldn’t spot any details on their estimates. They mention 70b+ params as being large in several places. But we’ve had several 100b+ param models that trail Sonnet.
Coffeezilla bought one of these thinking they’d never be delivered about a week before they announced they would be shipping soon. He wanted to do an exposé on the delays and thought Trump would never release the phone He will now end up with a crappy phone and his personal info exposed
reply