As a species with a hyper-powered neocortex, we seem to be burning bright lately (1). The pace of AI feels like an evolutionary leap—punctuated, surprising and relentless. What was great a month ago, is now legacy. We are being trained by the models into a new form of thinking and reasoning. As we iterate prompts, they teach us new question-formulation skills—forcing humans to give more than we receive. The upgrade has arrived: context engineering.
Prompt engineering served us well: we feed the model on average 50 input tokens in exchange for 500 output tokens. It seemed like a bargain. But Karpathy's advocacy for 'context engineering' is now rewriting how we conceptualize computational thinking. It is a lesson in generosity. A path towards a non-zero game. A new symmetry between what we give and what we get back.
We used to spend 10 seconds on a question and several months on the answer. What would it mean if this were to reverse? What if the infinitesimal (the question) becomes the infinite, and the answer becomes the infinitesimal that opens a portal to a dimension of lucid understanding? Imagine dumping the whole of physics in your context window to find E=mc², an epiphany that opens a portal of knowledge.
Kolmogorov Complexity and the Magic of Compression
This is where algorithmic information theory fits in. Kolmogorov complexity represents how "compressible" a piece of information is. If a string (e.g., a sequence of numbers) can only be described in its entirety, it has high complexity. Conversely, if it can be described concisely by a pattern or formula, it has low complexity.
The string "1111111111" has low complexity because it can be described succinctly as "ten 1's in a row."
The string "101101100101" has higher complexity because no simple pattern may describe it, requiring a longer program to recreate it exactly.
Simply put: the ultimate compression equals the best understanding. I have always found Kolmogorov complexity magical. Just as an incantation might transform a single word or phrase into vast and detailed results, Kolmogorov complexity finds the shortest possible way to regenerate an entire dataset, no matter how large or detailed it is.
It is also mysterious in a way. Kolmogorov complexity is uncomputable in general (halting problem vibes), but approximations via MDL guide AI toward that elusive simplicity.
There is something unreachably elusive about universal representation—capturing complexity with simplicity, and this feels mystical. In music, Kolmogorov complexity could quantify a riff's essence—compressing a symphony into its shortest algorithmic seed, much like AI distilling genres into generative code.
Rissanen's MDL principle is about finding the best way to explain a dataset by keeping things as simple as possible while still capturing all the important details. It balances two things: how simple the model is and how well it describes the data. The goal is to pick the model that uses the least amount of information to represent both the model and the data together. Think of it like summarizing a story— you want the shortest summary that still tells the whole tale accurately.
In Kolmogorov-Rissanen world, every answer that the AI gives you back, is the ultimate compression: the infinitesimal kernel that, once decoded, unveils the infinite. Context engineering is the art of blowing up your question and then compressing it down to its irreducible essence.
From THIN Questions to FAT Questions
We often invest weeks or months dissecting a problem, gathering data, iterating solutions, and polishing outputs. Conversely, the initial question or prompt is usually sketched in minutes—sometimes seconds—leaving ambiguity that gets resolved via trial-and-error. It's a betting game fueled by unearned dopamine and laziness.
We will start with a good system prompt that will define the lens through which the model reasons. Then we will use prompt backpropagation: we prompt the prompt itself and use AI to refine our question into the most information-rich, unambiguous form. E.g., start with 'Generate a jazz solo' → AI refines to 'Generate a 16-bar jazz solo in C minor at 120 BPM, emulating Miles Davis phrasing via MIDI output, with entropy constraints for improvisation feel.'
Like gradient steps on a loss surface, each AI-generated tweak nudges your prompt toward MDL. Achieving MDL from a prompt demands true understanding of both the domain and the model's capabilities—moving us from mere "memorizers" to genuine "grokkers." Context engineering is the best of both worlds: half art, half science. Left and right hemispheres are in sync.
Today: 1. Ask a THIN question. 2. Receive a FAT answer.
Tomorrow: 1. Ask a FAT question. 2. Receive a THIN answer.
Context engineering will raise the bar for domain literacy. (I had a math teacher who told us "if you cannot formulate the right question, you probably do not deserve the answer")
Implications for Music AI
Leading AI music generation platforms such as Suno and Udio, AIVA, Boomy, Soundverse, ACE STEP, Magenta Studio, Beatoven.ai, Loudly, Stable Audio, Musicfy, Sonauto, Soundraw, and Tad.ai, Tencent Music, ByteDance, NetEase Cloud Music, Kuaishou, all work with prompt engineering. You craft a prompt like "Create a 30-second EDM drop with a heavy bassline and 128 BPM," and the model generates based solely on this input. It does it fast, no setup is required; ideal for casual use. But the outputs lack depth or consistency, especially for complex tasks like multi-instrument arrangements, as the model relies only on the prompt's instructions.
But that is not the only constraint. For lack of a license to train on the music these models have ingested, and after being sued by RIAA, the lawyers have dramatically limited the admissible content in the context window. For instance, instead of in the style of the Beatles, you have to say, "young group from Liverpool from the sixties". Lawyers also set up massive guard rails (you cannot say "sounds like" when you target a copyrighted voice or song).
What happened to the greatest LMM ever? By injecting these constraints, models will degrade over time (remember what happened to the greatest large music model ever, Google's Orca (2) after the lawyers had their way with it). Google DeepMind shelved it late last year due to copyright feedback from labels, deeming it a "huge legal risk". (Unlike Suno or UDIO, Alphabet could pay the fines). The lawyers tried to make it legal by randomly pruning it, but the model collapsed. Post-Orca, models like Suno v4 now emphasize synthetic data to skirt lawsuits, but they run the same risks.
Context engineering is the next level in managing music AI. Let's say you compose a cohesive album, and you give the model a dataset of the artist's previous work for style consistency. Metadata specifying genres, instruments, and emotional arcs and a memory buffer retaining themes from earlier tracks in the album. This produces richer, more tailored outputs; maintains coherence. But it does require more setup (e.g., curating datasets) and computational resources, which can be challenging for users without technical expertise.
How to get your own dataset. The ideal way to go about it is to curate high-quality datasets, such as MIDI files, audio samples, or metadata, relevant to the task. See to it that the music is copyright-free or has a license. Royalty-free music is still copyrighted, but the user pays a one-time fee to use it without further royalties. Creative Commons licenses offer various usage permissions, some allowing free use with attribution. Music composed before 1923, is often in the public domain. This means it can be used without permission or payment. Open-source: Tools like InspireMusic (Alibaba's GitHub repo) let you fine-tune models on your datasets.
Use Hybrid Syntax. To get better results with context engineering use "hybrid syntax": combine natural language with structured elements (e.g., mixing descriptive text with formatted data like JSON, XML, YAML, or markdown). This helps the LLM parse complex information more effectively, reducing errors like hallucinations or irrelevant responses. For instance, instead of dumping raw text into the context, you can structure retrieved data (from databases, documents, or prior interactions) as JSON objects, which makes it easier for the model to reference specific fields.
If you're unsure about JSON syntax, leveraging the LLM itself (via a separate "prompt window" or chained prompt) to convert unstructured text into JSON is a practical workaround. This is a form of multi-step prompting or "chain-of-prompts," where you first instruct the model to reformat data (e.g., "Convert the following text to a valid JSON object with keys for 'name', 'description', and 'value'") before feeding it into the main context. This trick is implicit in many workflows, as it ensures clean, structured inputs without manual expertise, and it's commonly recommended for beginners or dynamic systems.
A Future Where Questions Are the Product
Here is a provocative future in which FAT questions (context engineering) become the true product and THIN answers the deliverable. All follow the same pattern: invest human capital in question design → offload execution to AI.
Consultancy as "Prompt Architects". Just imagine consultancy firms McKinsey & Company, Boston Consulting Group (BCG), Bain & Company, Deloitte, PwC, EY, and KPMG selling questions instead of answers. Their services would become products which you could buy online. Catalogs of modular prompts, organized by function (e.g., market entry, M&A diligence, digital transformation). Prompt blueprints that consist of sequences of interlocking prompts that guide an AI through multi-step analyses (data ingestion → hypothesis generation → risk modeling → recommendation). Roadmaps & playbooks built as prompts that the client runs end-to-end.
Record Labels as Context Curators: Majors like Universal sell 'style embeddings'—vectorized datasets of anonymized hits—for engineers to inject into models, ensuring coherence without infringement.
Amazon selling "Book Embeddings". I read a lot of books in my life, but in retrospect it is a big time inefficiency. How much do we really remember after reading a book? How many quotes do we retain in our sparse long term memory. It would have been great if I had a vector database of all the books I read with my comments and my summaries. That is now possible. Think of precomputed vector embeddings for every title, theme-level, equation, structured table. They could sell a "personal vector library" subscription: you aggregate these embeddings into your own private index which you can add to any AI you use as external memory. This would give you on-demand summaries & cross-book syntheses via vector queries. Amazon could also have bonus "tracks" like book-adjacent prompts which you buy alongside your e-book: e.g. a "30-day reading companion" prompt, or "case-study deep-dive" prompt.
This will turn Amazon (and all publishers) into data providers, unlocking new revenue streams beyond one-time book sales and readers, researchers, and corporations can rapidly surface and synthesize knowledge across thousands of titles. You buy one never-ending book.
Investigative Journalism. Investigative journalists could come back who spend weeks sculpting a single interview question. Once finalized, readers could hit a list of public figures (as characters) and simply hit "answer," and a comprehensive report auto-generates. Headlines would read, "After 3-month Question Only, President's 30-second Reply Reveals X."
Basic research has always been about questions, not answers. Universities award multi-year fellowships to scholars who devise that one seminal thesis question. Students spend their graduate careers refining their question; upon submission, AI instantly drafts the entire dissertation (literature review, methodology, conclusions).
Pharmaceutical R&D that makes clinical trials a formality. Drug companies no longer run years of trials—they labor for 18 months to formulate the precise "mechanism‐of‐action" question. Once posed, the AI instantly outlines a safe, effective molecule and dosage.
Law Firms Selling depositions. Litigation transforms into months of wrangling over the exact wording of a single interrogatory. When delivered, the AI immediately provides all case-law precedents, persuasive arguments, and settlement figures.
Marketing Prompt Studios. Teams of creatives iterate for months to craft one marketing question. On release, AI instantly produces slogan, visuals, segmentation strategy, and conversion forecasts.
Personal Development Retreats. Self-help companies run 10-day silent retreats where participants meditate and journal to unearth their one "life-purpose question." At the end, AI delivers a complete life-plan roadmap in minutes.
Scientific Breakthrough Auctions. Research labs auction off months of peer-reviewed question development ("What is the unifying theory between dark matter and quantum entanglement?"). The highest bidder poses it to the AI and instantly receives a detailed theoretical framework.
What will be the ultimate price of the most desired good, the question or the answer?
In the age of GenAI, true wisdom will not be about knowing the answer, but knowing how to ask the perfect question. This could reverse the price of input and output tokens.
A Larger Purpose
There is a larger purpose underlying our collective journey. We are in a punctuated equilibrium of progress and change always rattles the status quo. But let's not forget the initial hedonistic missions of our species: to replace the philosophy of need by the philosophy of want (Maslow) and to replace a life of doing by a life of thinking (Aristotle, "bios theoretikos").
After centuries of struggle and mechanization, we are nearly there. We stand on the brink of our last unfulfilled longing: to evolve beyond human limits. Either we invite AI into our brains as co-pilot, or relegate it to traffic controllers, preserving our solo flight over the abyss toward a transcendent new land. I believe we'll let it in—our children certainly will—but many may yet savor "the eternal sunshine of the spotless mind", free from algorithmic whispers.
The centre of me is always and eternally in terrible pain ... A searching for something beyond what the world contains, something transfiguring and infinite. (Bertrand Russell, 1872-1970)
References:
1. William Blake, Auguries of Innocence (1803).
2. Google DeepMind's AI Music Generator Raises Copyright Questions – Future Party.