Оффлайн
- Регистрация
- 1 Мар 2015
- Сообщения
- 1,481
- Баллы
- 155
This article is a translation of .
What is llms.txt?
llms.txt is a Markdown-formatted text file proposed to address the limitation that LLMs have small context windows and cannot process entire websites.
It was proposed on September 3, 2024, by Jeremy Howard of Answer.AI at . It is not defined in an RFC like robots.txt.
The format includes certain specifications such as the website name in an H1 section, a brief summary, and a list of links in H2 sections.
Example:
There's also llms-full.txt, which contains all website information.
Example:
There are directory sites for llms.txt as well:
I've added both:
The necessary work involves modifying or creating just three files:
I added the following to config.toml:
Defining output formats for llms and llmsfull while adding them to the default value of outputs.home1, which is ["html", "rss"].
[outputs]
home = ["html", "rss", "llms", "llmsfull"]
[outputFormats.llms]
baseName = "llms"
isPlainText = true
mediaType = "text/plain"
rel = "alternate"
root = true
[outputFormats.llmsfull]
baseName = "llms-full"
isPlainText = true
mediaType = "text/plain"
rel = "alternate"
root = true
layouts/index.llms.txt
I created layouts/index.llms.txt as a template for llms.txt:
# {{ .Site.Title }}
> {{ .Site.Params.Description }}
## Articles
{{ $yearMonthGroups := slice -}}
{{ range where (where (sort (.Site.GetPage "/posts/").Pages "Date" "desc") "Draft" "ne" true) "Sitemap.Disable" "ne" true -}}
{{ $yearMonth := .Date.Format "2006/01" -}}
{{ if not (in $yearMonthGroups $yearMonth) }}
* {{ $yearMonth -}}
{{ $yearMonthGroups = $yearMonthGroups | append $yearMonth -}}
{{ end }}
* [{{ .Title }}]({{ .Permalink }})
{{- end }}
## Others
{{- $baseURL := .Site.BaseURL | strings.TrimSuffix "/" }}
* [Device]({{ $baseURL }}/device/): 所有デバイス
* [History]({{ $baseURL }}/history/): このサイトの歴史
* [About]({{ $baseURL }}/about/): 自己紹介
* [Privacy Policy]({{ $baseURL }}/privacy/): 当サイトの広告、アフィリエイト、プライバシーポリシー
Since Hugo site structures can vary widely, this is just one example. My blog has both blog posts and other articles, and since other articles don't change much, I hardcoded them in the Others section.
layouts/index.llmsfull.txt
I created layouts/index.llmsfull.txt as a template for llms-full.txt:
# {{ .Site.Title }}
> {{ .Site.Params.Description }}
{{/* Articles */}}
{{ range where (where (sort (.Site.GetPage "/posts/").Pages "Date" "desc") "Draft" "ne" true) "Sitemap.Disable" "ne" true }}
--------------------------------------------------------------------------------
title: "{{ .Title }}"
date: "{{ .Date.Format "2006-01-02" }}"
--------------------------------------------------------------------------------
{{ replaceRE "{{<\\s*comment\\s*>}}(.|\n)*?{{<\\s*/comment\\s*>}}" "" .RawContent -}}
{{ end -}}
{{/* Others */}}
{{ range slice "device.md" "history.md" "about.md" "privacy.md" -}}
{{ with site.GetPage . -}}
--------------------------------------------------------------------------------
title: "{{ .Title }}"
lastmod: "{{ .Date.Format "2006-01-02" }}"
--------------------------------------------------------------------------------
{{ replaceRE "{{<\\s*comment\\s*>}}(.|\n)*?{{<\\s*/comment\\s*>}}" "" .RawContent -}}
{{ end -}}
{{ end -}}
I made the delimiter longer because some articles contained ---.
Also, I made sure comments like the following wouldn't be included in llms-full.txt:
{{<comment>}}
This is a Hugo comment. It won't be output as an HTML comment either.
{{</comment>}}
nginx Configuration
Since llms.txt and llms-full.txt were showing character encoding issues, I added the following to the server directive:
location ~ "^/llms(-full)?\.txt" {
root /usr/share/nginx/html;
charset UTF-8;
}
Conclusion
I added llms.txt and llms-full.txt on my Hugo-built blog:
Regardless of whether there's demand for it, I'm satisfied with the result.
As a side note, I initially found a method on that used resources.ExecuteAsTemplate in layouts/robots.txt to generate llms.txt. However, I decided not to use this approach since including llms.txt in robots.txt is not yet common practice.
Another side note: in the past, my blog was created by converting a single ChangeLog-formatted file into HTML using a tool called . Creating llms-full.txt reminded me of those days. It feels like I've come full circle.
What is llms.txt?
llms.txt is a Markdown-formatted text file proposed to address the limitation that LLMs have small context windows and cannot process entire websites.
It was proposed on September 3, 2024, by Jeremy Howard of Answer.AI at . It is not defined in an RFC like robots.txt.
The format includes certain specifications such as the website name in an H1 section, a brief summary, and a list of links in H2 sections.
Example:
There's also llms-full.txt, which contains all website information.
Example:
There are directory sites for llms.txt as well:
I've added both:
The necessary work involves modifying or creating just three files:
- config.toml (modify)
- layouts/index.llms.txt (create new)
- layouts/index.llmsfull.txt (create new)
I added the following to config.toml:
Defining output formats for llms and llmsfull while adding them to the default value of outputs.home1, which is ["html", "rss"].
[outputs]
home = ["html", "rss", "llms", "llmsfull"]
[outputFormats.llms]
baseName = "llms"
isPlainText = true
mediaType = "text/plain"
rel = "alternate"
root = true
[outputFormats.llmsfull]
baseName = "llms-full"
isPlainText = true
mediaType = "text/plain"
rel = "alternate"
root = true
layouts/index.llms.txt
I created layouts/index.llms.txt as a template for llms.txt:
# {{ .Site.Title }}
> {{ .Site.Params.Description }}
## Articles
{{ $yearMonthGroups := slice -}}
{{ range where (where (sort (.Site.GetPage "/posts/").Pages "Date" "desc") "Draft" "ne" true) "Sitemap.Disable" "ne" true -}}
{{ $yearMonth := .Date.Format "2006/01" -}}
{{ if not (in $yearMonthGroups $yearMonth) }}
* {{ $yearMonth -}}
{{ $yearMonthGroups = $yearMonthGroups | append $yearMonth -}}
{{ end }}
* [{{ .Title }}]({{ .Permalink }})
{{- end }}
## Others
{{- $baseURL := .Site.BaseURL | strings.TrimSuffix "/" }}
* [Device]({{ $baseURL }}/device/): 所有デバイス
* [History]({{ $baseURL }}/history/): このサイトの歴史
* [About]({{ $baseURL }}/about/): 自己紹介
* [Privacy Policy]({{ $baseURL }}/privacy/): 当サイトの広告、アフィリエイト、プライバシーポリシー
Since Hugo site structures can vary widely, this is just one example. My blog has both blog posts and other articles, and since other articles don't change much, I hardcoded them in the Others section.
layouts/index.llmsfull.txt
I created layouts/index.llmsfull.txt as a template for llms-full.txt:
# {{ .Site.Title }}
> {{ .Site.Params.Description }}
{{/* Articles */}}
{{ range where (where (sort (.Site.GetPage "/posts/").Pages "Date" "desc") "Draft" "ne" true) "Sitemap.Disable" "ne" true }}
--------------------------------------------------------------------------------
title: "{{ .Title }}"
date: "{{ .Date.Format "2006-01-02" }}"
--------------------------------------------------------------------------------
{{ replaceRE "{{<\\s*comment\\s*>}}(.|\n)*?{{<\\s*/comment\\s*>}}" "" .RawContent -}}
{{ end -}}
{{/* Others */}}
{{ range slice "device.md" "history.md" "about.md" "privacy.md" -}}
{{ with site.GetPage . -}}
--------------------------------------------------------------------------------
title: "{{ .Title }}"
lastmod: "{{ .Date.Format "2006-01-02" }}"
--------------------------------------------------------------------------------
{{ replaceRE "{{<\\s*comment\\s*>}}(.|\n)*?{{<\\s*/comment\\s*>}}" "" .RawContent -}}
{{ end -}}
{{ end -}}
I made the delimiter longer because some articles contained ---.
Also, I made sure comments like the following wouldn't be included in llms-full.txt:
{{<comment>}}
This is a Hugo comment. It won't be output as an HTML comment either.
{{</comment>}}
nginx Configuration
Since llms.txt and llms-full.txt were showing character encoding issues, I added the following to the server directive:
location ~ "^/llms(-full)?\.txt" {
root /usr/share/nginx/html;
charset UTF-8;
}
Conclusion
I added llms.txt and llms-full.txt on my Hugo-built blog:
Regardless of whether there's demand for it, I'm satisfied with the result.
As a side note, I initially found a method on that used resources.ExecuteAsTemplate in layouts/robots.txt to generate llms.txt. However, I decided not to use this approach since including llms.txt in robots.txt is not yet common practice.
Another side note: in the past, my blog was created by converting a single ChangeLog-formatted file into HTML using a tool called . Creating llms-full.txt reminded me of those days. It feels like I've come full circle.
