Theory of Constraints Introduction

[Dface]

@Max

LLMs need an ‘alphabet’ to take in information.

I see alphabet is in single quotes so I’m thinking u’re using that word as a metaphor? Or as a lack of better words.

We use alphabets to build words that we can talk/write with. AI maybe does the same with the tokens.

Ok, mb I didn’t read the rest of the quote above after the word alphabet. To take information in sounds interesting. When i give the AI a prompt I’m thinking it takes information in via the tokens. Like, maybe when I tell it to make up a short paragraph of a short story, it uses AI tokens to take information from my prompt? Like, maybe because I said paragraph it can do something related to a paragraph? Depends on the tokens. I’m not sure.

@Max

English isn’t very good because it has so few symbols, which are inefficient to run an LLM on.

I think you’re talking about the alphabet? Those symbols? Imma look it up

I saw maybe u mean stuff like, “#$%^.” There doesn’t seem to be a lot of those symbols. I wonder how many symbols an LLM needs and why. I also see that u actually say that it’s inefficient to run an LLM on English. I bet it takes so long to run it on English.

@Max

Instead, we could create an alphabet with like 200,000 characters which is much more efficient.

I bet the LLM can access those 200,000 characters quickly cuz that’s a lot of characters. I wonder why it’s more efficient.

@Max

Practically, this (very roughly) equates to 4/3 tokens per word.

So maybe there’s tokens with parts of words. Maybe it could be “anti” and “thesis” from the word antithesis.

@Max

Some words are one token, other words are made up of multiple tokens.

Maybe the word(if we’re talking about English words) paragraph is made the tokens, “para” and “graph”. Maybe the tokens are unrelated to the actual words? Like, maybe the “many” token and “sentences” token can express the word paragraph? Idk

@Max

LLMs have something called a context window which is the most amount of information they can ‘see’ at once.

I wonder if information can be measured. Like is it quantitative? If we’re talking about bits from a computer then maybe yeah?

I looked up the question via gemini, “what’s the most information in can a context window have?” and I got this:

It seems information is measured in tokens.

@Max

You can’t fit more data in, but you can summarize data to ‘compress’ it.

I wonder if u can be creative to compress the data.

@Max

This context window is measured in tokens – the largest context windows today are about 1M tokens (so about 750k words).

Mb maybe i should read everything first quickly then do a careful reading?

Seems like u can convert tokens to words. They’re both information. It’s like converting miles to kilometers maybe?

@Max

For comparison, all 3 LOTR books together are about 470k words.

more than half a million tokens i think.

@Max

If you want to see what it looks like, check out https://platform.openai.com/tokenizer (paste in any text)

ty ill take a look. Maybe it all this will help with the AI token constraint examples(unburden the constraint).

I tried a word and look at that. It takes three tokens not two:

@Max

Technically, each time you send a message to an LLM you are sending the entire chat history with your most recent message at the end. So in a sense you can think of the whole chat history as ‘the prompt’ that the AI completes.

I was wondering how the AI tracks the convo. That’s if it takes the whole convo history as a prompt. Im wondering if u keep messaging it and add to the prompt, does the AI change its answer? Like, would it respond differently to something u said before? Like, it changed its “opinion” along the way.