How Qwen models work: tokens, embeddings, attention and scaling laws

A Qwen model reads text as tokens, turns each token into a vector, mixes the vectors through stacked attention layers, and ends with a probability for every token it knows. It writes one token at a time and repeats. Chapter 3 of Generative AI on AWS explains each step and says you can build applications without the details. That is true until the invoice arrives. On the Qwen tokenizer the same instruction is 10 tokens in English and 20 in Thai, and the context window, the attention cost and the size of your search index all follow from the steps on this page.

This is part 3 of the series. Part 1 covers setup and model choice. Part 2 covers prompting.

What chapter 3 covers

The chapter walks through how a large language model is built: model hubs and model cards, tokenizers, embedding vectors, the transformer and self-attention, the three transformer model types, pretraining datasets, scaling laws and compute-optimal models. Its examples use Llama 2, Falcon, FLAN-T5, BERT and BloombergGPT.

This page keeps the order and swaps in Qwen and Model Studio. Each section answers one practical question: what you are billed for, what you can store, what limits the model, or how to read a model’s size.

The chapter opens with model cards, the page a model hub publishes for each model. In Model Studio the equivalent is the model reference page, such as the one for qwen3.8-flash. Read five things there before you test a model: the context length and maximum output, whether thinking is on by default, the regions and deployment scopes that offer the model, the price per tier, and whether the ID is an alias or a dated snapshot. Each of those has changed an evaluation result or a bill in the first two parts of this series.

The book also warns against swapping to the newest model before you finish evaluating the current one. The Qwen release pace makes that warning stronger. Between April and August 2026 the family moved through 3.6, 3.7 and 3.8. Finish the test set on one model ID, record the score, and then compare the new release against that score.

Token counts on this page were measured on 8 October 2026 with the Qwen tokenizer file shipped in the DashScope Python SDK 1.27.7. The demo reproduces them. Hosted models can use a newer tokenizer, so treat local counts as estimates and the usage field of a real response as the billed figure.

Tokenizers turn text into billed units

A tokenizer splits text into pieces from a fixed vocabulary and replaces each piece with an integer ID. The model never sees letters. It sees IDs.

Qwen uses byte-pair encoding. Training starts from bytes and repeatedly merges the most frequent adjacent pair into a new vocabulary entry, so common words become one token and rare strings break into several. The tokenizer file in the DashScope SDK has 151,643 merged entries and 208 special tokens, 151,851 entries in total. The book cites 30,000 to 100,000 tokens as typical in 2023. Qwen’s vocabulary is larger, which keeps more words and more scripts as single tokens.

Measured splits:

The sentence on the first row maps to the IDs 785, 10652, 28405, 700, 1573, 15786, 9442. A leading space is part of the token, so " cluster" and "cluster" are different IDs.

The book’s rule of thumb is about 1.3 tokens per English word. It holds for plain English on Qwen. The 13-word support ticket used in part 1 measured 17 tokens, 1.31 per word. It fails for identifiers. Region names, version strings, JSON keys and log lines split into many short pieces. A JSON record of 59 characters measured 27 tokens, while a SQL statement of the same length measured 15. Estimate prompts full of IDs, hashes or JSON from samples of your own data.

The same instruction in five languages

One instruction, “Please summarize the incident report in three bullet points”, written in five languages:

Model Studio bills per token, so the language of the text is a price multiplier. A support assistant that reads 300 tokens of English per message pays USD 45 a month for one million messages on qwen3.8-flash input. The same content in Thai at 2.0x is about 600 tokens and USD 90 (calculated). Output tokens scale the same way at the higher output price.

What to do with this:

• Budget per language. Use a separate token-per-message figure for each language you serve, measured from real traffic.
• Write instructions in the cheapest language that passes your tests. The system message repeats on every call. User text stays in the user’s language.
• Check the hosted tokenizer. Qwen3.5 and later models were released with broader language coverage. Run tokens_billed.py from the demo. If the billed counts for Thai drop relative to the local estimate, the hosted model uses a different vocabulary and your budget can shrink.

Embeddings turn meaning into numbers

An embedding is a list of numbers that places a piece of text in a space where related meanings sit close together. The book shows this with a three-dimensional picture. Real embeddings have hundreds or thousands of dimensions.

There are two kinds to keep apart:

• The embedding layer inside Qwen. Each token ID maps to a learned vector. These vectors are internal to the model and you never receive them from the chat API.
• The embedding API. text-embedding-v4 takes whole passages and returns one vector per passage. You store those vectors and compare them to find related text. This is the search step of retrieval-augmented generation.

Model Studio’s text embedding models, as documented on 8 October 2026:

Similarity between two embeddings is usually measured with cosine similarity: 1 means the same direction, 0 means unrelated. Search works by embedding every document chunk once, storing the vectors, embedding each question at query time, and returning the chunks with the highest similarity.

Dimension choice is a storage and memory decision:

Index structures add memory on top of the raw vectors. Fewer dimensions keep less detail, and the loss shows up as lower recall on questions that need fine distinctions. The demo compares similarity scores at 1,024 and 256 dimensions. Run the same comparison on your own questions before you cut dimensions, and plan to re-embed the whole corpus if you change model or dimension, because vectors from different settings cannot be compared.

On Alibaba Cloud the vectors can live in AnalyticDB for PostgreSQL, Alibaba Cloud Elasticsearch, Hologres or OpenSearch Vector Search Edition, or inside a Model Studio knowledge base that manages them for you. For cluster sizing and engine tuning on the Elasticsearch and OpenSearch side, see the OpenSearch and Elasticsearch engineering guides at SearchStack Pro. Dense and sparse output types are available through the DashScope API only, using the output_type parameter.

Inside the transformer

The transformer is the architecture behind every model in this series. The book walks through it bottom to top. The diagram follows the same path for one prompt.

Input and context window

The context window is the maximum number of tokens the model handles in one request, input and output together. Everything counts: system message, examples, history, retrieved documents, the user’s message, any reasoning trace and the answer.

Embedding layer

Each token ID is replaced with its learned vector. From here on the model works with vectors only.

Self-attention

Self-attention lets each token draw information from other tokens in the sequence. For each token the model computes three vectors from learned weights: a query, a key and a value. The score between two tokens is the dot product of one token’s query and the other’s key. A softmax turns the scores into weights, and the token’s new representation is the weighted average of the value vectors. The book’s example has the word “her” attending strongly to “professor” earlier in the sentence.

Models run many attention heads in parallel, each with its own weights, so different heads can track different relationships. In a decoder-only model a token can attend only to tokens before it, which is what allows generation one token at a time.

Full attention compares every token with every other token, so its work grows with the square of the sequence length. Doubling a prompt roughly quadruples that part of the computation. Newer Qwen models reduce this. Qwen3.5 interleaves three Gated DeltaNet linear-attention layers with one full-attention layer, about 75% linear and 25% full, according to published architecture descriptions. Long prompts still cost more time and more input tokens, and Model Studio prices some models higher above 256K input tokens. qwen3.7-plus moves from USD 0.40 to USD 1.20 per million input tokens past that tier in Singapore.

Softmax output

The last layer produces one score per vocabulary entry, and a softmax converts the scores into probabilities that sum to 1. For a vocabulary of about 151,851 entries, that is about 151,851 probabilities for every generated token. The sampling parameters from part 2 act on this distribution. The chosen token is appended to the input and the model runs again, until it emits an end token or reaches max_completion_tokens.

Context windows: capacity and attention

qwen3.8-flash has these limits in Model Studio, checked on 8 October 2026:

The book’s figures were 512 tokens for FLAN-T5, 2,048 for Falcon and 4,096 for Llama 2. Capacity grew by more than 200 times. The book’s caution about it still applies. The Lost in the Middle study (Liu and others, 2023) found that models used information at the start and end of a long input more reliably than information in the middle. The size of the effect differs between models and versions, so test the model you deploy.

Use a large window for jobs that need the whole document at once, such as comparing two contracts or reviewing a full log file. For question answering over a document set, retrieval usually sends a few thousand relevant tokens instead of a million, which is cheaper per call and easier to test. Before relying on a long context, place a known fact at the start, middle and end of a long input and check that the model answers questions about each position.

Three model types and where Qwen fits

The book describes three kinds of transformer models by how they are trained.

The table describes the job each model does in a system. Alibaba Cloud does not publish the internal architecture of text-embedding-v4, so the last column describes the role each model plays.

For system design the split is simple. Use a decoder-only chat model to generate text. Use an embedding model to find text. Use a reranker to reorder the top search results before they reach the chat model.

Total and active parameters

The book sizes models by parameter count: 7 billion, 70 billion, 175 billion. Most large Qwen models now use a mixture-of-experts design. Each layer holds many expert sub-networks and a router sends each token to a few of them. Two numbers describe such a model:

Active parameters drive the compute per token, which is why a 397B model can answer about as fast as a 17B dense model. Total parameters drive memory, because every expert must be loaded. At 2 bytes per parameter in bf16, Qwen3.5-397B-A17B needs about 794 GB just for its weights (calculated), before any working memory. That figure matters the day you move from the Model Studio API to self-hosting on PAI-EAS. Chapter 4 of the book, and the next part of this series, cover memory and quantization.

On Model Studio none of this is your problem. You pay per token and Alibaba Cloud runs the hardware.

Pretraining data and scaling laws

Pretraining is the stage where a model learns language from a very large text corpus before any task tuning. The book covers the data sources, such as Common Crawl, Wikipedia, C4, The Pile and RefinedWeb, and the scaling laws that link model size, data size and compute.

The Chinchilla paper from 2022 found that for a fixed compute budget, a model should see about 20 training tokens per parameter. Models trained on less are larger than they need to be. The book notes that Llama 2 70B went past this ratio, with 2 trillion tokens, about 29 per parameter (calculated), and outperformed the original LLaMA.

The trend continued. Qwen3 was pretrained on about 36 trillion tokens. For the 235B flagship that is about 153 tokens per total parameter and about 1,636 per active parameter (calculated). Training this far past the compute-optimal ratio costs more up front and produces a smaller model for a given quality, which is cheaper to serve on every request afterwards.

Two conclusions for model choice:

• Parameter count is a weak quality signal. A heavily trained smaller model can beat a larger one on your task. Let the test set from part 1 decide, and ignore the size in the model name.
• Pretraining your own foundation model is out of scope for almost every team. The book’s BloombergGPT example used 1.3 million GPU hours. Start from a hosted or open-weight Qwen model and adapt it, which later parts of this series cover.

Demo: count tokens and compare embeddings

The demo has three scripts. The first runs on your laptop with no API key and no network call. The second and third call Model Studio in Singapore.

Before you start

Use the virtual environment from part 1 and add the DashScope SDK, which contains the Qwen tokenizer:

source .venv/bin/activate
pip install -U dashscope tiktoken openai
export DASHSCOPE_API_KEY="sk-ws-paste-your-key"
export DASHSCOPE_BASE_URL="https://dashscope-intl.aliyuncs.com/compatible-mode/v1"

Add text-embedding-v4 to your API key’s allowed models. It has its own free quota of 1,000,000 tokens in Singapore, and the embedding script uses well under 100 tokens per run.

Step 1: Count tokens locally

Save as tokens_local.py:

import sys

from dashscope import get_tokenizer

tokenizer = get_tokenizer("qwen-turbo")

SAMPLES = {
    "English": "Please summarize the incident report in three bullet points.",
    "German": "Bitte fassen Sie den Vorfallbericht in drei Stichpunkten zusammen.",
    "Chinese": "请用三个要点总结这份事故报告。",
    "Japanese": "インシデントレポートを3つの箇条書きで要約してください。",
    "Thai": "โปรดสรุปรายงานเหตุการณ์เป็นสามหัวข้อย่อย",
}


def show(text):
    ids = tokenizer.encode(text)
    pieces = [tokenizer.decode([i]) for i in ids]
    print(f"{len(ids):>3} tokens  {text}")
    print(f"     ids    {ids}")
    print(f"     pieces {pieces}")


if len(sys.argv) > 1:
    show(" ".join(sys.argv[1:]))
    sys.exit(0)

show("The cluster scales out before lunch traffic")
for word in ["kubectl", "Kubernetes", "FinOps", "ap-southeast-1", "Qwen3.8-Flash"]:
    print(f"{len(tokenizer.encode(word)):>3} tokens  {word}")

print("\nSame instruction, five languages")
base = len(tokenizer.encode(SAMPLES["English"]))
for language, text in SAMPLES.items():
    n = len(tokenizer.encode(text))
    print(f"{language:<9} {n:>3} tokens  {n / base:.1f}x English")

Run it:

python tokens_local.py

Output measured on 8 October 2026 with DashScope SDK 1.27.7:

  7 tokens  The cluster scales out before lunch traffic
     ids    [785, 10652, 28405, 700, 1573, 15786, 9442]
     pieces ['The', ' cluster', ' scales', ' out', ' before', ' lunch', ' traffic']
  1 tokens  kubectl
  2 tokens  Kubernetes
  2 tokens  FinOps
  6 tokens  ap-southeast-1
  7 tokens  Qwen3.8-Flash

Same instruction, five languages
English    10 tokens  1.0x English
German     17 tokens  1.7x English
Chinese     9 tokens  0.9x English
Japanese   18 tokens  1.8x English
Thai       20 tokens  2.0x English

Pass your own text as arguments to see how it splits:

python tokens_local.py "SELECT order_id FROM orders WHERE status = 'late';"

Step 2: Compare the estimate with the bill

Save as tokens_billed.py:

import os
import sys

from dashscope import get_tokenizer
from openai import OpenAI

BASE_URL = os.getenv("DASHSCOPE_BASE_URL", "https://dashscope-intl.aliyuncs.com/compatible-mode/v1")
MODEL = os.getenv("QWEN_MODEL", "qwen3.8-flash")

if not os.getenv("DASHSCOPE_API_KEY"):
    sys.exit("Set DASHSCOPE_API_KEY first.")

client = OpenAI(api_key=os.environ["DASHSCOPE_API_KEY"], base_url=BASE_URL)
tokenizer = get_tokenizer("qwen-turbo")

SYSTEM = "Reply with the single word OK."
PROMPTS = [
    "Please summarize the incident report in three bullet points.",
    "โปรดสรุปรายงานเหตุการณ์เป็นสามหัวข้อย่อย",
    "请用三个要点总结这份事故报告。",
]

print(f"{'local':>5} {'billed':>6} {'template':>8}  prompt")
for text in PROMPTS:
    local = len(tokenizer.encode(SYSTEM)) + len(tokenizer.encode(text))
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],
        max_completion_tokens=5,
        extra_body={"enable_thinking": False},
    )
    billed = resp.usage.prompt_tokens
    print(f"{local:>5} {billed:>6} {billed - local:>8}  {text}")

Run it:

python tokens_billed.py

The output has this shape:

local billed template  prompt
   17    <n>      <n>  Please summarize the incident report in three bullet points.
   27    <n>      <n>  โปรดสรุปรายงานเหตุการณ์เป็นสามหัวข้อย่อย
   16    <n>      <n>  请用三个要点总结这份事故报告。

The local column is fixed by the tokenizer and matches the measured values above. The billed column comes from Model Studio. The template column is the chat template overhead. If it is the same on all three rows, the hosted model uses the same vocabulary as the local file. If it changes by language, the hosted tokenizer differs and the billed figure is the one to budget with.

Step 3: Compare embeddings at two sizes

Save as embeddings_similarity.py:

import math
import os
import sys

from openai import OpenAI

BASE_URL = os.getenv("DASHSCOPE_BASE_URL", "https://dashscope-intl.aliyuncs.com/compatible-mode/v1")

if not os.getenv("DASHSCOPE_API_KEY"):
    sys.exit("Set DASHSCOPE_API_KEY first.")

client = OpenAI(api_key=os.environ["DASHSCOPE_API_KEY"], base_url=BASE_URL)

# Ten inputs: the documented maximum per text-embedding-v4 request.
TERMS = ["teach", "lecture", "student", "textbook", "autoscaler",
         "node pool", "Kubernetes", "invoice", "refund", "billing"]
PAIRS = [("teach", "lecture"), ("student", "textbook"), ("autoscaler", "node pool"),
         ("invoice", "refund"), ("teach", "invoice"), ("Kubernetes", "refund")]


def embed(dimensions):
    resp = client.embeddings.create(model="text-embedding-v4", input=TERMS, dimensions=dimensions, encoding_format="float")
    vectors = {TERMS[item.index]: item.embedding for item in resp.data}
    return vectors, resp.usage.total_tokens


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))


results = {}
for dims in (1024, 256):
    vectors, tokens = embed(dims)
    results[dims] = {pair: cosine(vectors[pair[0]], vectors[pair[1]]) for pair in PAIRS}
    print(f"dimensions={dims} vector length={len(next(iter(vectors.values())))} tokens billed={tokens}")

print(f"\n{'pair':<28}{'1024 dims':>10}{'256 dims':>10}")
for pair in PAIRS:
    print(f"{pair[0] + ' / ' + pair[1]:<28}{results[1024][pair]:>10.3f}{results[256][pair]:>10.3f}")

Run it:

python embeddings_similarity.py

The first four pairs are related terms and the last two are unrelated. In the output, the related pairs should score higher than the unrelated pairs at both sizes. If they do, and the same holds on a sample of your own questions and documents, 256 dimensions stores a quarter of the bytes for a similar ranking. If the gap between related and unrelated pairs shrinks a lot at 256, keep 1,024.

Planning FAQ

Why does the bill show more input tokens than my text?

Three reasons. The chat template adds role markers to every message. Conversation history is sent again on every turn. Reasoning traces count as input when you keep them in history. Step 2 of the demo measures the template part.

Can I count tokens before sending a request?

Yes, as an estimate. The tokenizer in the DashScope SDK runs locally and is exact for models that share its vocabulary. For other models, call once and read usage.prompt_tokens.

Which embedding dimension should a new index use?

Start with the 1,024 default. Test 512 and 256 on a labelled set of your own questions, and choose the smallest size whose recall meets your target. Index memory and storage scale with the dimension.

Do I need to re-embed documents when the embedding model changes?

Yes. Vectors from different models or different dimensions are not comparable. Store the model name and dimension with every vector so a migration can find what needs re-embedding.

Alibaba Cloud architecture consulting covers token budgets, embedding storage and vector database sizing for retrieval systems on Alibaba Cloud. Token counts on this page were measured on 8 October 2026. Model limits and prices were checked against the Alibaba Cloud documentation listed below on the same date.