In AI, tokenization is splitting text into small units called tokens, such as words or word pieces, which a language model reads and generates. Token counts set how much text fits in a model's context window and usually drive the cost of each API call.
Embeddings are lists of numbers that represent the meaning of text, images or other data, arranged so that similar items sit close together. They let software search, group and compare by meaning instead of exact words.
A large language model (LLM) is a neural network trained on very large amounts of text to predict and generate language. It powers chat assistants, summarisers, coding tools and AI agents.
A transformer is a neural network architecture that uses attention to weigh the relationships between all parts of an input. It is the basis of most modern language models.
A context window is the maximum amount of text, counted in tokens, that a language model can take into account in one request. It covers the instructions, the conversation, any retrieved documents and the reply.
A clear proven process, that delivers
You've seen our ideas - now go deeper.
Discover more insights, guides, and engineering practice.