The Wayward Axolotl

To deal with complexity, we need to analyze things into components, and hide information inside those components. The black-box metaphor is common in science and engineering. A black box is a system with a well-defined and well-understood interface. The contents of the system are either hidden or can be ignored. We can think about the system only in terms of its “surface”: the inputs and outputs. Hence the term “black box”. In science and engineering, we call things “black boxes” if either (a) we don’t know how they work, or (b) we choose to ignore how they work, usually to simplify reasoning. For example, a black-box optimization algorithm has two inputs: a space S and a black-box function U : S → R. The algorithm has no access to the internal details of U. It can only apply U to points in S. From this information, the algorithm tries to find a point in the space that has the highest value of U, within time constraints. Any black-box optimization algorithm must sample ...

This is a little discussion I had with Claude, the LLM agent created by Anthropic. My prompts are in block-quotes. The rest of the text is from Claude. For more on the nature of knowledge, see Theories of Knowledge and Knowledge and Reality . ✦ ✦ ✦ Explain how the principle of information compression relates to LLMs. Information Compression and LLMs The relationship is deep and arguably foundational to what LLMs are. The core idea Information compression, in the technical sense, is about finding compact representations that preserve the essential structure of data while discarding redundancy. A good compressor must learn the underlying patterns — the grammar, regularities, and dependencies — of the data it’s compressing. LLMs are, in a meaningful sense, learned compressors of human language and knowledge. Training as compression When an LLM is trained on a vast corpus of text, it’s forced to distill billions of documents into a fi...

Talking with Ethan Howell about knowledge. Moderated by Eric Claussen . Relevant links: Knowledge and Reality Theories of Knowledge Claude Explains Representationalism

Talking with Garrick about primitivism and other things. Essays mentioned: Modern Civilization is Irrational Fires, Polar Bears and Global Warming