The current llm.txt file in @apps/portal/llm.txt is intended to provide information for Large Language Models (LLMs) but is currently just a list of URLs. This does not conform to the emerging llms.txt specification https://llmstxt.org/, which defines a structured Markdown file for LLM content indexing.
This ticket aims to correctly implement the llms.txt specification for the Neo.mjs website and clarify its purpose in relation to a potential llm.txt for crawler permissions.
Acceptance Criteria:
- Rename/Refactor
llm.txt to llms.txt: The existing llm.txt in @apps/portal/llm.txt should be replaced or transformed into a new llms.txt file that adheres to the https://llmstxt.org/ specification.
- Update
generateSeoFiles.mjs: The buildScripts/generateSeoFiles.mjs script must be modified to generate the new llms.txt file.
llms.txt Content Structure: The generated llms.txt must:
- Be a Markdown document.
- Include a mandatory H1 header for the project/site name (e.g.,
# Neo.mjs Platform).
- Contain an optional blockquote for a concise summary of the website.
- Feature additional Markdown sections for detailed information.
- Include H2-delimited "file lists" with URLs pointing to more detailed Markdown versions of relevant pages (e.g., key guides, API documentation).
- Content Generation Strategy: Define and implement a strategy for populating the
llms.txt content. This could involve:
- Automatically extracting summaries from existing Markdown files.
- Allowing for manual curation of key summaries and links.
- Leveraging the
learn/tree.json structure to identify important content.
- Differentiation: Clarify the role of
llm.txt (for crawler permissions) versus llms.txt (for content indexing). If a separate llm.txt for permissions is deemed necessary and distinct from robots.txt, a plan for its generation should be included. For now, the primary focus is on llms.txt.
The current
llm.txtfile in@apps/portal/llm.txtis intended to provide information for Large Language Models (LLMs) but is currently just a list of URLs. This does not conform to the emergingllms.txtspecification https://llmstxt.org/, which defines a structured Markdown file for LLM content indexing.This ticket aims to correctly implement the
llms.txtspecification for the Neo.mjs website and clarify its purpose in relation to a potentialllm.txtfor crawler permissions.Acceptance Criteria:
llm.txttollms.txt: The existingllm.txtin@apps/portal/llm.txtshould be replaced or transformed into a newllms.txtfile that adheres to thehttps://llmstxt.org/specification.generateSeoFiles.mjs: ThebuildScripts/generateSeoFiles.mjsscript must be modified to generate the newllms.txtfile.llms.txtContent Structure: The generatedllms.txtmust:# Neo.mjs Platform).llms.txtcontent. This could involve:learn/tree.jsonstructure to identify important content.llm.txt(for crawler permissions) versusllms.txt(for content indexing). If a separatellm.txtfor permissions is deemed necessary and distinct fromrobots.txt, a plan for its generation should be included. For now, the primary focus is onllms.txt.