How Computers Learned to Read: From Words to Numbers

It is a pleasant morning. You open your apartment food-order app and type:

"Please bring one hot cup of tea with a little extra milk."

To the kitchen staff, the request is clear. They know what you want, prepare the order, and send it to your home.

Now consider a computer.

A computer has no eyes, no ears, and no real-world intuition. At its core, it is still a very fast calculator that understands only numbers—specifically \(0\)s and \(1\)s.

How does a system take a normal sentence like "Please bring one hot cup of tea with a little extra milk" and understand what it means?

It breaks down the problem into three simple steps:

Step 1: Text to Tokens

A computer cannot swallow an entire sentence in one go. First, it passes your order through a digital shredder called a Tokenizer.

The tokenizer chops the sentence into small building blocks called tokens. A token can be a full word, a syllable, or even punctuation:

Why chop words into small pieces?

• If you say "cup" or "cups", the core idea remains identical.
• Breaking words into sub-word tokens allows the computer to easily recognise spelling mistakes, local slang, and fresh terms without maintaining an unending dictionary.

Step 2: Tokens to Token IDs

Even after chopping, the machine still cannot interpret letters like “t-e-a”.

To solve this, the model refers to a master catalogue (the model's vocabulary list). Think of it like giving a roll number to every unique token.

Now, your order is converted into a neat sequence of numbers:

At this stage, the computer can store your sentence in memory. However, to the system, tea (7415) and milk (9182) are still just arbitrary numbers without any clear connection.

Step 3: Embeddings

To understand that tea is a hot morning drink and not a motor scooter or a laptop, the computer plots the tokens onto a coordinate graph. In AI, these coordinates are called Embeddings.

Note: While this diagram uses two axes for simplicity, real-world language models map tokens across hundreds or thousands of abstract numerical dimensions to capture complex semantic relationships.

Step 4: Context Alters the Position

What happens if you modify the order slightly?

"Bhaiya, give one iced cup of tea..."

The model's internal attention mechanism spots the qualifier "iced" sitting right next to "tea". In real-world language models, embeddings consist of hundreds of dimensions, where individual coordinates capture specific semantic attributes such as temperature. Thus, when "iced" is added, the coordinate value corresponding to temperature flips from positive (hot) to negative (cold)—shifting the vector representation from [0.9, 0.2, -0.45, ...] towards [-0.8, 0.2, -0.45, ...] to reflect the updated context, bridging the 2D conceptual diagram with multi-dimensional vector math.

This enables the system to adapt its understanding based on surrounding context.

Step 5: How the AI Formulates a Reply

Once the context-adjusted vector (such as the updated coordinates for "iced tea" from Step 4) is finalized, it serves as the exact starting input for the model's autoregressive response generation:

1. It checks the collective position of your prompt.
2. It reviews its training patterns to find what logically follows this context.
3. It generates the response token by token:

What's Next?

In our next post, we will explore What is an LLM and its core fundamentals.