v0.9.7

- postpone building until really needed
- rebuild only if artifacts change
- no build on source update
- prune takes current resolver state instead of global var

.gitattributes

@@ -0,0 +1 @@
+examples/** linguist-documentation

.gitignore

@@ -1,5 +1,5 @@
 .DS_Store
-/dist-env/
+/env-publish/
 
 __pycache__/
 *.py[cod]

Makefile

@@ -1,21 +1,16 @@
-.PHONY: help
+dist: setup.py lektor_groupby/*
-help:
-	@echo 'commands:'
-	@echo '  dist'
-
-dist-env:
-	@echo Creating virtual environment...
-	@python3 -m venv 'dist-env'
-	@source dist-env/bin/activate && pip install twine
-
-.PHONY: dist
-dist: dist-env
-	[ -z "$${VIRTUAL_ENV}" ]  # you can not do this inside a virtual environment.
-	rm -rf dist
 	@echo Building...
 	python3 setup.py sdist bdist_wheel
-	@echo
 	rm -rf ./*.egg-info/ ./build/ MANIFEST
+
+env-publish:
+	@echo Creating virtual environment...
+	@python3 -m venv 'env-publish'
+	@source env-publish/bin/activate && pip install twine
+
+.PHONY: publish
+publish: dist env-publish
+	[ -z "$${VIRTUAL_ENV}" ]  # you can not do this inside a virtual environment.
 	@echo Publishing...
-	@echo "\033[0;31mEnter your PyPI token:\033[0m"
+	@echo "\033[0;31mEnter PyPI token in password prompt:\033[0m"
-	@source dist-env/bin/activate && export TWINE_USERNAME='__token__' && twine upload dist/*
+	@source env-publish/bin/activate && export TWINE_USERNAME='__token__' && twine upload dist/*

README.md

@@ -1,168 +1,26 @@
 # Lektor Plugin: groupby
 
-A generic grouping / clustering plugin. Can be used for tagging or similar tasks.
+A generic grouping / clustering plugin.
+Can be used for tagging or similar tasks.
+The grouping algorithm is performed once.
+Contrary to, at least, cubic runtime if doing the same with Pad queries.
 
-Overview:
+Install this plugin or modify your Lektor project file:
-- the [basic example](#usage-basic-example) goes into detail how this plugin works.
-- the [quick config](#usage-quick-config) example show how you can use the plugin config to setup a quick and easy tagging system.
-- the [complex example](#usage-a-slightly-more-complex-example) touches on the potential of what is possible.
 
+```sh
+lektor plugin add groupby
+```
 
-## Usage: Basic example
+Optionally, enable a basic config:
-
-Lets start with a simple example: adding a tags field to your model.
-Assuming you have a `blog-entry.ini` that is used for all children of `/blog` path.
-
-
-#### `models/blog-entry.ini`
 
 ```ini
-[fields.tags]
+[tags]
-label = Tags
+root = /
-type = strings
+slug = tag/{group}.html
-myvar = true
+template = tag.html
-
-[fields.body]
-label = Content
-type = markdown
-```
-
-Notice we introduce a new attribute variable: `myvar = true`.
-The name can be anything here, we will come to that later.
-The only thing that matters is that the value is a boolean and set to true.
-
-Edit your blog entry and add these two new tags:
-
-```
-Awesome
-Latest News
-```
-
-Next, we need a plugin to add the groupby event listener.
-
-
-#### `packages/test/lektor_my_tags_plugin.py`
-
-```python
-def on_groupby_init(self, groupby, **extra):
-    @groupby.watch('/blog', 'myvar', flatten=True, template='myvar.html',
-                   slug='tag/{group}/index.html')
-    def do_myvar(args):
-        page = args.record  # extract additional info from source
-        fieldKey, flowIndex, flowKey = args.key  # or get field index directly
-        # val = page.get(fieldKey).blocks[flowIndex].get(flowKey)
-        value = args.field  # list type since model is 'strings' type
-        for tag in value:
-            yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
-```
-
-There are a few important things here:
-
-1. The first parameter (`'/blog'`) is the root page of the groupby.
-   All results will be placed under this directory, e.g., `/blog/tags/clean/`.
-   You can also just use `/`, in which case the same path would be `/tags/clean/`.
-   Or create multiple listeners, one for `/blog/` and another for `/projects/`, etc.
-2. The second parameter (`'myvar'`) must be the same attribute variable we used in our `blog-entry.ini` model.
-   The groupby plugin will traverse all models and search for this attribute name.
-3. Flatten determines how Flow elements are processed.
-   If `False`, the callback function `convert_myvar()` is called once per Flow element (if the Flow element has the `myvar` attribute attached).
-   If `True` (default), the callback is called for all Flow blocks individually.
-4. The template `myvar.html` is used to render the grouping page.
-   This parameter is optional.
-   If no explicit template is set, the default template `groupby-myvar.html` would be used. Where `myvar` is replaced with whatever attribute you chose.
-5. Finally, the slug `tag/{group}/index.html` is where the result is placed.
-   The default value for this parameter is `{attrib}/{group}/index.html`.
-   In our case, the default path would resolve to `myvar/awesome/index.html`.
-   We explicitly chose to replace the default slug with our own, which ignores the attrib path component and instead puts the result pages inside the `/tag` directory.
-   (PS: you could also use for example `t/{group}.html`, etc.)
-
-
-So much for the `args` parameter.
-The callback body **can** produce groupings but does not have to.
-If you choose to produce an entry, you have to `yield` a tuple pair of `(groupkey, extra-info)`.
-`groupkey` is used to combine & cluster pages and must be URL-safe.
-The `extra-info` is passed through to your template file.
-You can yield more than one entry per source or filter / ignore pages if you don't yield anything.
-Our simple example will generate the output files `tag/awesome/index.html` and `tag/latest-news/index.html`.
-
-Lets take a look at the html next.
-
-
-#### `templates/myvar.html`
-
-```html
-<h2>Path: {{ this | url(absolute=True) }}</h2>
-<div>This is: {{this}}</div>
-<ul>
-	{%- for child in this.children %}
-	<li>Page: {{ child.record.path }}, Name: {{ child.extra.val }}, Tag count: {{ child.extra.tags_in_page }}</li>
-	{%- endfor %}
-</ul>
-```
-
-Notice, we can use `child.record` to access the referenced page of the group cluster.
-`child.extra` contains the additional information we previously passed into the template.
-
-The final result of `tag/latest-news/index.html`:
-
-```
-Path: /tag/latest-news/
-This is: <GroupBySource attribute="myvar" group="latest-news" template="myvar.html" slug="tag/latest-news/" children=1>
-  -  Page: /blog/barss, Name: Latest News, Tag count: 2
-```
-
-
-## Usage: Quick config
-
-The whole example above can be simplified with a plugin config:
-
-#### `configs/groupby.ini`
-
-```ini
-[myvar]
-root = /blog/
-slug = tag/{group}/index.html
-template = myvar.html
 split = ' '
 ```
 
-You still need to add a separate attribute to your model (step 1), but anything else is handled by the config file.
+Or dive into plugin development...
-All of these fields are optional and fallback to the default values stated above.
 
-The newly introduced option `split` will be used as string delimiter.
+For usage examples, refer to the [examples](https://github.com/relikd/lektor-groupby-plugin/tree/main/examples) readme.
-This allows to have a field with `string` type instead of `strings` type.
-If you do not provide the `split` option, the whole field value will be used as group key.
-Note: split is only used on str fields (`string` type), not lists (`strings` type).
-
-The emitted `extra-info` for the child is the original key value.
-E.g., `Latest News,Awesome` with `split = ,` yields `('latest-news', 'Latest News')` and `('awesome', 'Awesome')`.
-
-
-## Usage: A slightly more complex example
-
-There are situations though, where a simple config file is not enough.
-The following plugin will find all model fields with attribute `inlinetags` and search for in-text occurrences of `{{Tagname}}` etc.
-
-```python
-from lektor.markdown import Markdown
-from lektor.types.formats import MarkdownDescriptor
-from lektor.utils import slugify
-import re
-_regex = re.compile(r'{{([^}]{1,32})}}')
-
-def on_groupby_init(self, groupby, **extra):
-    @groupby.watch('/', 'inlinetags', slug='tags/{group}/')
-    def convert_inlinetags(args):
-        arr = args.field if isinstance(args.field, list) else [args.field]
-        for obj in arr:
-            if isinstance(obj, (Markdown, MarkdownDescriptor)):
-                obj = obj.source
-            if isinstance(obj, str) and str:
-                for match in _regex.finditer(obj):
-                    tag = match.group(1)
-                    yield slugify(tag), tag
-```
-
-This generic approach does not care what data-type the field value is:
-`strings` fields will be expanded and enumerated, Markdown will be unpacked.
-You can combine this mere tag-detector with text-replacements to point to the actual tags-page.

examples/Examples.lektorproject

@@ -0,0 +1,5 @@
+[project]
+name = GroupBy Examples
+
+[packages]
+lektor-groupby = 0.9.7

examples/Makefile

@@ -0,0 +1,7 @@
+.PHONY: server clean plugins
+server:
+	lektor server
+clean:
+	lektor clean --yes -v
+plugins:
+	lektor plugins flush-cache && lektor plugins list

examples/README.md

@@ -0,0 +1,292 @@
+# Usage
+
+Overview:
+- [quick config example](#quick-config) shows how you can use the plugin config to setup a quick and easy tagging system.
+- [simple example](#simple-example) goes into detail how to use it in your own plugin.
+- [advanced example](#advanced-example) touches on the potentials of the plugin.
+- [Misc](#misc) shows other use-cases.
+
+After reading this tutorial, have a look at other plugins that use `lektor-groupby`:
+- [lektor-inlinetags](https://github.com/relikd/lektor-inlinetags-plugin)
+
+
+## About
+
+To use the groupby plugin you have to add an attribute to your model file.
+In our case you can refer to the [`models/page.ini`](./models/page.ini) model:
+
+```ini
+[fields.tags]
+label = Tags
+type = strings
+testA = true
+testB = true
+
+[fields.body]
+label = Body
+type = markdown
+testC = true
+```
+
+We did define three custom attributes `testA`, `testB`, and `testC`.
+You may add custom attributes to all of the fields.
+It is crucial that the value of the custom attribute is set to true.
+The attribute name is later used for grouping.
+
+
+
+## Quick config
+
+Relevant files:
+
+- [`configs/groupby.ini`](./configs/groupby.ini)
+- [`templates/example-config.html`](./templates/example-config.html)
+
+
+The easiest way to add tags to your site is by defining the `groupby.ini` config file.
+
+```ini
+[testA]
+root = /
+slug = config/{key}.html
+template = example-config.html
+split = ' '
+enabled = True
+
+[testA.fields]
+title = "Tagged: " ~ this.group
+
+[testA.key_map]
+Blog = News
+```
+
+The configuration parameter are:
+
+1. The section title (`testA`) must be one of the attribute variables we defined in our model.
+2. The `root` parameter (`/`) is the root page of the groupby.
+   All results will be placed under this directory, e.g., `/tags/tagname/`.
+   If you use `root = /blog`, the results path will be `/blog/tags/tagname/`.
+   The groupby plugin will traverse all sub-pages wich contain the attribute `testA`.
+3. The `slug` parameter (`config/{key}.html`) is where the results are placed.
+   In our case, the path resolves to `config/tagname.html`.
+   The default value is `{attrib}/{key}/index.html` which would resolve to `testA/tagname/index.html`.
+   If this field contains `{key}`, it just replaces the value with the group-key.
+   In all other cases the field value is evaluated in a jinja context.
+4. The `template`parameter (`example-config.html`) is used to render the results page.
+   If no explicit template is set, the default template `groupby-testA.html` will be used.
+   Where `testA` is replaced with whatever attribute you chose.
+5. The `split` parameter (`' '`) will be used as string delimiter.
+   Fields of type `strings` and `checkboxes` are already lists and don't need splitting.
+   The split is only relevant for fields of type `string` or `text`.
+   These single-line fields are then expanded to lists as well.
+   If you do not provide the `split` option, the whole field value will be used as tagname.
+6. The `enabled` parameter allows you to quickly disable the grouping.
+
+
+You can have multiple listeners, e.g., one for `/blog/` and another for `/projects/`.
+Just create as many custom attributes as you like, each having its own section.
+
+There are two additional config mappings, `.fields` and `.key_map`.
+Key-value pairs in `.fields` will be added as attributes to your grouping.
+You can access them in your template (e.g., `{{this.title}}`).
+All of the `.fields` values are evaluted in a jinja context, so be cautious when using plain strings. 
+
+The built-in field attributes are:
+
+- `group`: returned group name, e.g., "A Title?"
+- `key`: slugified group value, e.g., "a-title"
+- `slug`: url path after root node, e.g. "config/a-title.html" (can be `None`)
+- `record`: parent node, e.g., `Page(path="/")`
+- `children`: dictionary of `{record: extras}` pairs
+- `first_child`: first page
+- `first_extra`: first extra
+- `config`: configuration object (see below)
+
+Without any changes, the `key` value will just be `slugify(group)`.
+However, the other mapping `.key_map` will replace `group` with whatever replacement value is provided in the `.key_map` and then slugified.
+You could, for example, add a `C# = c-sharp` mapping, which would otherwise just be slugified to `c`.
+This is equivalent to `slugify(key_map.get(group))`.
+
+The `config` attribute contains the values that created the group:
+
+- `key`: attribute key, e.g., `TestA`
+- `root`: as provided by init, e.g., `/`
+- `slug`: the raw value, e.g., `config/{key}.html`
+- `template`: as provided by init, e.g., `example-config.html`
+- `enabled`: boolean
+- `dependencies`: path to config file (if initialized from config)
+- `fields`: raw values from `TestA.fields`
+- `key_map`: raw values from `TestA.key_map`
+
+In your template file you have access to the attributes, config, and children (pages):
+
+```jinja2
+<h2>{{ this.title }}</h2>
+<p>Key: {{ this.key }}, Attribute: {{ this.config.key }}</p>
+<ul>
+{%- for child in this.children %}
+<li>Page: {{ child.path }}</li>
+{%- endfor %}
+</ul>
+```
+
+
+
+## Simple example
+
+Relevant files:
+
+- [`packages/simple-example/lektor_simple.py`](./packages/simple-example/lektor_simple.py)
+- [`templates/example-simple.html`](./templates/example-simple.html)
+
+
+```python
+def on_groupby_before_build_all(self, groupby, builder, **extra):
+    watcher = groupby.add_watcher('testB', {
+        'root': '/blog',
+        'slug': 'simple/{key}/index.html',
+        'template': 'example-simple.html',
+    })
+    watcher.config.set_key_map({'Foo': 'bar'})
+    watcher.config.set_fields({'date': datetime.now()})
+
+    @watcher.grouping(flatten=True)
+    def convert_simple_example(args):
+        # Yield groups
+        value = args.field  # type: list # since model is 'strings' type
+        for tag in value:
+            yield tag, {'tags_in_page': value}
+```
+
+This example is roughly equivalent to the config example above – the parameters of the `groupby.add_watcher` function correspond to the same config parameters.
+Additionally, you can set other types in `set_fields` (all strings are evaluated in jinja context!).
+
+`@watcher.grouping` sets the callback to generate group keys.
+It has one optional flatten parameter:
+
+- `flatten` determines how Flow elements are processed.
+  If `False`, the callback function is called once per Flow element.
+  If `True` (default), the callback is called for all Flow-Blocks of the Flow individually.
+  The attribute `testB` can be attached to either the Flow or a Flow-Block regardless.
+
+The `args` parameter of the `convert_simple_example()` function is a named tuple, it has three attributes:
+
+1. The `record` points to the `Page` record that contains the tag.
+2. The `key` tuple `(field-key, flow-index, flow-key)` tells which field is processed.
+   For Flow types, `flow-index` and `flow-key` are set, otherwise they are `None`.
+3. The `field` value is the content of the processed field.
+   The field value is equivalent to the following:
+
+```python
+k = args.key
+field = args.record[k.fieldKey].blocks[k.flowIndex].get(k.flowKey)
+```
+
+The callback body **can** produce groupings but does not have to.
+If you choose to produce an entry, you have to `yield` a string or tuple pair `(group, extra-info)`.
+`group` is slugified (see above) and then used to combine & cluster pages.
+The `extra-info` (optional) is passed through to your template file.
+You can yield more than one entry per source.
+Or ignore pages if you don't yield anything.
+
+The template file can access and display the `extra-info`:
+
+```jinja2
+<p>Custom field date: {{this.date}}</p>
+<ul>
+{%- for child, extras in this.children.items() -%}
+{%- set etxra = (extras|first).tags_in_page %}
+<li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
+{%- endfor %}
+</ul>
+```
+
+
+
+## Advanced example
+
+Relevant files:
+
+- [`configs/advanced.ini`](./configs/advanced.ini)
+- [`packages/advanced-example/lektor_advanced.py`](./packages/advanced-example/lektor_advanced.py)
+- [`templates/example-advanced.html`](./templates/example-advanced.html)
+
+
+The following example is similar to the previous one.
+Except that it loads a config file and replaces in-text occurrences of `{{Tagname}}` with `<a href="/tag/">Tagname</a>`.
+
+```python
+def on_groupby_before_build_all(self, groupby, builder, **extra):
+    # load config
+    config = self.get_config()
+    regex = config.get('testC.pattern.match')
+    try:
+        regex = re.compile(regex)
+    except Exception as e:
+        print('inlinetags.regex not valid: ' + str(e))
+        return
+
+    # load config directly (which also tracks dependency)
+    watcher = groupby.add_watcher('testC', config)
+
+    @watcher.grouping()
+    def convert_replace_example(args):
+        # args.field assumed to be Markdown
+        obj = args.field.source
+        slugify_map = {}  # type Dict[str, str]
+        for match in regex.finditer(obj):
+            tag = match.group(1)
+            key = yield tag
+            print('[advanced] slugify:', tag, '->', key)
+            slugify_map[tag] = key
+
+        def _fn(match: re.Match) -> str:
+            tag = match.group(1)
+            return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
+        args.field.source = regex.sub(_fn, obj)
+```
+
+Notice, `add_watcher` accepts a config file as parameter which keeps also track of dependencies and rebuilds pages when you edit the config file.
+Further, the `yield` call returns the slugified group-key.
+First, you do not need to slugify it yourself and second, potential replacements from `key_map` are already handled.
+
+For Markdown fields, we can modify the `source` attribute directly.
+All other field types need to be accessed via `args.record` key indirection (see [simple example](#simple-example)).
+
+```ini
+[testC]
+root = /
+slug = "advanced/{}/".format(this.key)
+template = example-advanced.html
+
+[testC.pattern]
+match = {{([^}]{1,32})}}
+```
+
+The config file takes the same parameters as the [config example](#quick-config).
+As you can see, `slug` is evaluated in jinja context.
+
+We introduced a new config option `testC.pattern.match`.
+This regular expression matches `{{` + any string less than 32 characters + `}}`.
+Notice, the parenthesis (`()`) will match only the inner part but the replace function (`re.sub`) will remove the `{{}}`.
+
+
+
+## Misc
+
+It was shortly mentioned above that slugs can be `None` (only if manually set to `slug = None`).
+This is useful if you do not want to create subpages but rather an index page containing all groups.
+This can be done in combination with the next use-case:
+
+```jinja2
+{%- for x in this|vgroups(keys=['TestA', 'TestB'], fields=[], flows=[], recursive=True)|unique|sort %} 
+<a href="{{ x|url }}">({{ x.group }})</a>
+{%- endfor %} 
+```
+
+You can query the groups of any parent node (including those without slug).
+[`templates/page.html`](./templates/page.html) uses this.
+The keys (`'TestA', 'TestB'`) can be omitted which will return all groups of all attributes (you can still filter them with `x.config.key == 'TestC'`).
+The `fields` and `flows` params are also optional.
+With these you can match groups in `args.key.fieldKey` and `args.key.flowKey`.
+For example, if you have a “tags” field and an “additional-tags” field and you only want to show one in a preview.

examples/configs/advanced.ini

@@ -0,0 +1,15 @@
+[testC]
+root = /
+slug = "advanced/{}/".format(this.key)
+template = example-advanced.html
+
+[testC.pattern]
+match = {{([^}]{1,32})}}
+
+[testC.fields]
+desc = "Tag: " ~ this.group ~ ", Key: " ~ this.key
+
+[testC.key_map]
+Blog = case-sensitive
+Two = three
+three = no-nested-replace

examples/configs/groupby.ini

@@ -0,0 +1,12 @@
+[testA]
+enabled = True
+root = /
+slug = config/{key}.html
+template = example-config.html
+split = ' '
+
+[testA.fields]
+title = "Tagged: " ~ this.group
+
+[testA.key_map]
+Blog = News

examples/content/blog/contents.lr

@@ -0,0 +1,9 @@
+_model: blog
+---
+title: Blog
+---
+tags:
+
+Directory
+Blog
+samegroup

examples/content/blog/first-post/contents.lr

@@ -0,0 +1,9 @@
+title: Hello Website
+---
+body: This is an example blog post. {{Tag}} in {{blog}}
+---
+tags:
+
+Initial
+Blog Post
+samegroup

examples/content/contents.lr

@@ -0,0 +1,8 @@
+title: GroupBy Examples
+---
+body: Main body {{tag}} {{Two}}
+---
+tags:
+
+Root
+samegroup

examples/content/projects/contents.lr

@@ -0,0 +1,9 @@
+title: Projects
+---
+body:
+
+This is a list of the projects:
+
+* Project 1
+* Project 2
+* Project 3

examples/models/blog-post.ini

@@ -0,0 +1,4 @@
+[model]
+name = Blog Post
+inherits = page
+hidden = yes

examples/models/blog.ini

@@ -0,0 +1,7 @@
+[model]
+name = Blog
+inherits = page
+hidden = yes
+
+[children]
+model = blog-post

examples/models/page.ini

@@ -0,0 +1,18 @@
+[model]
+name = Page
+label = {{ this.title }}
+
+[fields.title]
+label = Title
+type = string
+
+[fields.tags]
+label = Tags
+type = strings
+testA = true
+testB = true
+
+[fields.body]
+label = Body
+type = markdown
+testC = true

examples/packages/advanced-example/lektor_advanced.py

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+from lektor.pluginsystem import Plugin
+from typing import Generator
+import re
+from lektor_groupby import GroupBy, GroupByCallbackArgs
+
+
+class AdvancedGroupByPlugin(Plugin):
+    def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
+        # load config
+        config = self.get_config()
+        regex = config.get('testC.pattern.match')
+        try:
+            regex = re.compile(regex)
+        except Exception as e:
+            print('inlinetags.regex not valid: ' + str(e))
+            return
+
+        # load config directly (which also tracks dependency)
+        watcher = groupby.add_watcher('testC', config)
+
+        @watcher.grouping()
+        def _replace(args: GroupByCallbackArgs) -> Generator[str, str, None]:
+            # args.field assumed to be Markdown
+            obj = args.field.source
+            slugify_map = {}  # type Dict[str, str]
+            for match in regex.finditer(obj):
+                tag = match.group(1)
+                key = yield tag
+                print('[advanced] slugify:', tag, '->', key)
+                slugify_map[tag] = key
+
+            def _fn(match: re.Match) -> str:
+                tag = match.group(1)
+                return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
+            args.field.source = regex.sub(_fn, obj)

examples/packages/advanced-example/setup.py

@@ -0,0 +1,12 @@
+from setuptools import setup
+
+setup(
+    name='lektor-advanced',
+    py_modules=['lektor_advanced'],
+    version='1.0',
+    entry_points={
+        'lektor.plugins': [
+            'advanced = lektor_advanced:AdvancedGroupByPlugin',
+        ]
+    }
+)

examples/packages/simple-example/lektor_simple.py

@@ -0,0 +1,33 @@
+# -*- coding: utf-8 -*-
+from lektor.pluginsystem import Plugin
+from typing import Iterator, Tuple
+from datetime import datetime
+from lektor_groupby import GroupBy, GroupByCallbackArgs
+
+
+class SimpleGroupByPlugin(Plugin):
+    def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
+        watcher = groupby.add_watcher('testB', {
+            'root': '/blog',
+            'slug': 'simple/{key}/index.html',
+            'template': 'example-simple.html',
+        })
+        watcher.config.set_key_map({'Foo': 'bar'})
+        watcher.config.set_fields({'date': datetime.now()})
+
+        @watcher.grouping(flatten=True)
+        def fn_simple(args: GroupByCallbackArgs) -> Iterator[Tuple[str, dict]]:
+            # Yield groups
+            value = args.field  # type: list # since model is 'strings' type
+            for tag in value:
+                yield tag, {'tags_in_page': value}
+            # Everything below is just for documentation purposes
+            page = args.record  # extract additional info from source
+            fieldKey, flowIndex, flowKey = args.key  # or get field index
+            if flowIndex is None:
+                obj = page[fieldKey]
+            else:
+                obj = page[fieldKey].blocks[flowIndex].get(flowKey)
+            print('[simple] page:', page)
+            print('[simple]  obj:', obj)
+            print('[simple] ')

examples/packages/simple-example/setup.py

@@ -0,0 +1,12 @@
+from setuptools import setup
+
+setup(
+    name='lektor-simple',
+    py_modules=['lektor_simple'],
+    version='1.0',
+    entry_points={
+        'lektor.plugins': [
+            'simple = lektor_simple:SimpleGroupByPlugin',
+        ]
+    }
+)

examples/templates/blog-post.html

@@ -0,0 +1,4 @@
+{% extends "page.html" %}
+{% block body %}
+  {{ this.body }}
+{% endblock %}

examples/templates/blog.html

@@ -0,0 +1,6 @@
+{% extends "page.html" %}
+{% block body %}
+  {% for child in this.children %}
+    <a href="{{child|url}}">{{ child.title }}</a>
+  {% endfor %}
+{% endblock %}

examples/templates/example-advanced.html

@@ -0,0 +1,4 @@
+<h2>Path: {{ this | url(absolute=True) }}</h2>
+<p>This is: {{this}}</p>
+<p>Custom field, desc: "{{this.desc}}"</p>
+<p>Children: {{this.children}}</p>

examples/templates/example-config.html

@@ -0,0 +1,9 @@
+<h2>Path: {{ this | url(absolute=True) }}</h2>
+<p>This is: {{this}}</p>
+<p>Group: "{{this.group}}", Key: "{{this.key}}"</p>
+<p>Custom field title: {{this.title}}</p>
+<ul>
+{%- for child in this.children %}
+<li>Child: <a href="{{child|url}}">{{child.title}}</a> ({{child.path}})</li>
+{%- endfor %}
+</ul>

examples/templates/example-simple.html

@@ -0,0 +1,9 @@
+<h2>Path: {{ this | url(absolute=True) }}</h2>
+<p>This is: {{this}}</p>
+<p>Custom field date: {{this.date}}</p>
+<ul>
+{%- for child, extras in this.children.items() -%}
+{%- set etxra = (extras|first).tags_in_page %}
+<li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
+{%- endfor %}
+</ul>

examples/templates/page.html

@@ -0,0 +1,29 @@
+<!doctype html>
+<meta charset="utf-8">
+<title>{% block title %}Welcome{% endblock %}</title>
+<style type="text/css">
+header, footer { padding: 1em; background: #DDD; }
+main { margin: 3em; }
+</style>
+<body>
+  <header>
+    <div>Nav: &nbsp;
+      <a href="{{ '/'|url }}">Root</a> &nbsp;
+      <a href="{{ '/blog'|url }}">Blog</a> &nbsp;
+      <a href="{{ '/projects'|url }}">Projects</a>
+    </div>
+  </header>
+  <main>
+    <h2>{{ this.title }}</h2>
+    {% block body %}{{ this.body }}{% endblock %}
+  </main>
+  <footer>
+    {%- for k, v in [('testA','Config'),('testB','Simple'),('testC','Advanced')] %}
+    <div>{{v}} Tags:
+      {%- for x in this|vgroups(k, recursive=True)|unique|sort %} 
+        <a href="{{ x|url }}">({{x.key}})</a>
+      {%- endfor %} 
+    </div>
+    {%- endfor %}
+  </footer>
+</body>

lektor_groupby.py

@@ -1,478 +0,0 @@
-# -*- coding: utf-8 -*-
-import lektor.db  # typing
-from lektor.build_programs import BuildProgram
-from lektor.builder import Artifact, Builder  # typing
-from lektor.pluginsystem import Plugin
-from lektor.reporter import reporter
-from lektor.sourceobj import SourceObject, VirtualSourceObject
-from lektor.types.flow import Flow, FlowType
-from lektor.utils import bool_from_string, build_url, prune_file_and_folder
-# for quick config
-from lektor.utils import slugify
-
-from typing import \
-    NewType, NamedTuple, Tuple, Dict, Set, List, Optional, Iterator, Callable
-
-VPATH = '@groupby'  # potentially unsafe. All matching entries are pruned.
-
-
-# -----------------------------------
-#            Typing
-# -----------------------------------
-FieldValue = NewType('FieldValue', object)  # lektor model data-field value
-AttributeKey = NewType('AttributeKey', str)  # attribute of lektor model
-GroupKey = NewType('GroupKey', str)  # key of group-by
-
-
-class FieldKeyPath(NamedTuple):
-    fieldKey: str
-    flowIndex: Optional[int] = None
-    flowKey: Optional[str] = None
-
-
-class GroupByCallbackArgs(NamedTuple):
-    record: lektor.db.Record
-    key: FieldKeyPath
-    field: FieldValue
-
-
-class GroupByCallbackYield(NamedTuple):
-    key: GroupKey
-    extra: object
-
-
-GroupingCallback = Callable[[GroupByCallbackArgs],
-                            Iterator[GroupByCallbackYield]]
-
-
-class GroupProducer(NamedTuple):
-    attribute: AttributeKey
-    func: GroupingCallback
-    flatten: bool = True
-    slug: Optional[str] = None
-    template: Optional[str] = None
-    dependency: Optional[str] = None
-
-
-class GroupComponent(NamedTuple):
-    record: lektor.db.Record
-    extra: object
-
-
-# -----------------------------------
-#            Actual logic
-# -----------------------------------
-
-
-class GroupBySource(VirtualSourceObject):
-    '''
-    Holds information for a single group/cluster.
-    This object is accessible in your template file.
-    Attributes: record, attribute, group, children, template, slug
-
-    :DEFAULTS:
-    template: "groupby-attribute.html"
-    slug: "{attrib}/{group}/index.html"
-    '''
-
-    def __init__(
-        self,
-        record: lektor.db.Record,
-        attribute: AttributeKey,
-        group: GroupKey,
-        children: List[GroupComponent] = [],
-        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
-        template: Optional[str] = None,  # default: "groupby-attribute.html"
-        dependency: Optional[str] = None
-    ):
-        super().__init__(record)
-        self.attribute = attribute
-        self.group = group
-        self.children = children
-        self.template = template or 'groupby-{}.html'.format(self.attribute)
-        self.dependency = dependency
-        # custom user path
-        slug = slug or '{attrib}/{group}/index.html'
-        slug = slug.replace('{attrib}', self.attribute)
-        slug = slug.replace('{group}', self.group)
-        if slug.endswith('/index.html'):
-            slug = slug[:-10]
-        self.slug = slug
-
-    @property
-    def path(self) -> str:
-        # Used in VirtualSourceInfo, used to prune VirtualObjects
-        return build_url([self.record.path, VPATH, self.attribute, self.group])
-
-    @property
-    def url_path(self) -> str:
-        return build_url([self.record.path, self.slug])
-
-    def iter_source_filenames(self) -> Iterator[str]:
-        if self.dependency:
-            yield self.dependency
-        for record, _ in self.children:
-            yield from record.iter_source_filenames()
-
-    def __str__(self) -> str:
-        txt = '<GroupBySource'
-        for x in ['attribute', 'group', 'template', 'slug']:
-            txt += ' {}="{}"'.format(x, getattr(self, x))
-        return txt + ' children={}>'.format(len(self.children))
-
-
-class GroupByBuildProgram(BuildProgram):
-    ''' Generates Build-Artifacts and write files. '''
-
-    def produce_artifacts(self) -> None:
-        url = self.source.url_path
-        if url.endswith('/'):
-            url += 'index.html'
-        self.declare_artifact(url, sources=list(
-            self.source.iter_source_filenames()))
-        GroupByPruner.track(url)
-
-    def build_artifact(self, artifact: Artifact) -> None:
-        self.source.pad.db.track_record_dependency(self.source)
-        artifact.render_template_into(self.source.template, this=self.source)
-
-
-# -----------------------------------
-#              Helper
-# -----------------------------------
-
-
-class GroupByPruner:
-    '''
-    Static collector for build-artifact urls.
-    All non-tracked VPATH-urls will be pruned after build.
-    '''
-    _cache: Set[str] = set()
-    # Note: this var is static or otherwise two instances of
-    #       GroupByCreator would prune each others artifacts.
-
-    @classmethod
-    def track(cls, url: str) -> None:
-        cls._cache.add(url.lstrip('/'))
-
-    @classmethod
-    def prune(cls, builder: Builder) -> None:
-        ''' Remove previously generated, unreferenced Artifacts. '''
-        dest_path = builder.destination_path
-        con = builder.connect_to_database()
-        try:
-            with builder.new_build_state() as build_state:
-                for url, file in build_state.iter_artifacts():
-                    if url.lstrip('/') in cls._cache:
-                        continue  # generated in this build-run
-                    infos = build_state.get_artifact_dependency_infos(url, [])
-                    for v_path, _ in infos:
-                        if VPATH not in v_path:
-                            continue  # we only care about groupby Virtuals
-                        reporter.report_pruned_artifact(url)
-                        prune_file_and_folder(file.filename, dest_path)
-                        build_state.remove_artifact(url)
-                        break  # there is only one VPATH-entry per source
-        finally:
-            con.close()
-        cls._cache.clear()
-
-
-# -----------------------------------
-#           Main Component
-# -----------------------------------
-
-
-class GroupByCreator:
-    '''
-    Process all children with matching conditions under specified page.
-    Creates a grouping of pages with similar (self-defined) attributes.
-    The grouping is performed only once per build (or manually invoked).
-    '''
-
-    def __init__(self):
-        self._flows: Dict[AttributeKey, Dict[str, Set[str]]] = {}
-        self._models: Dict[AttributeKey, Dict[str, Dict[str, str]]] = {}
-        self._func: Dict[str, Set[GroupProducer]] = {}
-        self._resolve_map: Dict[str, GroupBySource] = {}  # only for server
-        self._watched_once: Set[GroupingCallback] = set()
-
-    # --------------
-    #   Initialize
-    # --------------
-
-    def initialize(self, db: lektor.db):
-        self._flows.clear()
-        self._models.clear()
-        self._resolve_map.clear()
-        for prod_list in self._func.values():
-            for producer in prod_list:
-                self._register(db, producer.attribute)
-
-    def _register(self, db: lektor.db, attrib: AttributeKey) -> None:
-        ''' Preparation: find models and flow-models which contain attrib '''
-        if attrib in self._flows or attrib in self._models:
-            return  # already added
-        # find flow blocks with attrib
-        _flows = {}  # Dict[str, Set[str]]
-        for key, flow in db.flowblocks.items():
-            tmp1 = set(f.name for f in flow.fields
-                       if bool_from_string(f.options.get(attrib, False)))
-            if tmp1:
-                _flows[key] = tmp1
-        # find models with attrib or flow-blocks containing attrib
-        _models = {}  # Dict[str, Dict[str, str]]
-        for key, model in db.datamodels.items():
-            tmp2 = {}  # Dict[str, str]
-            for field in model.fields:
-                if bool_from_string(field.options.get(attrib, False)):
-                    tmp2[field.name] = '*'  # include all children
-                elif isinstance(field.type, FlowType):
-                    if any(x in _flows for x in field.type.flow_blocks):
-                        tmp2[field.name] = '?'  # only some flow blocks
-            if tmp2:
-                _models[key] = tmp2
-
-        self._flows[attrib] = _flows
-        self._models[attrib] = _models
-
-    # ----------------
-    #   Add Observer
-    # ----------------
-
-    def watch(
-        self,
-        root: str,
-        attrib: AttributeKey, *,
-        flatten: bool = True,  # if False, dont explode FlowType
-        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
-        template: Optional[str] = None,  # default: "groupby-attrib.html"
-        dependency: Optional[str] = None
-    ) -> Callable[[GroupingCallback], None]:
-        '''
-        Decorator to subscribe to attrib-elements. Converter for groupby().
-        Refer to groupby() for further details.
-
-        (record, field-key, field) -> (group-key, extra-info)
-
-        :DEFAULTS:
-        template: "groupby-attrib.html"
-        slug: "{attrib}/{group}/index.html"
-        '''
-        root = root.rstrip('/') + '/'
-
-        def _decorator(fn: GroupingCallback):
-            if root not in self._func:
-                self._func[root] = set()
-            self._func[root].add(
-                GroupProducer(attrib, fn, flatten, template, slug, dependency))
-
-        return _decorator
-
-    def watch_once(self, *args, **kwarg) -> Callable[[GroupingCallback], None]:
-        ''' Same as watch() but listener is auto removed after build. '''
-        def _decorator(fn: GroupingCallback):
-            self._watched_once.add(fn)
-            self.watch(*args, **kwarg)(fn)
-        return _decorator
-
-    def remove_watch_once(self) -> None:
-        ''' Remove all watch-once listeners. '''
-        for k, v in self._func.items():
-            not_once = {x for x in v if x.func not in self._watched_once}
-            self._func[k] = not_once
-        self._watched_once.clear()
-
-    # ----------
-    #   Helper
-    # ----------
-
-    def iter_record_fields(
-        self,
-        source: lektor.db.Record,
-        attrib: AttributeKey,
-        flatten: bool = False
-    ) -> Iterator[Tuple[FieldKeyPath, FieldValue]]:
-        ''' Enumerate all fields of a lektor.db.Record with attrib = True '''
-        assert isinstance(source, lektor.db.Record)
-        _flows = self._flows.get(attrib, {})
-        _models = self._models.get(attrib, {})
-
-        for r_key, subs in _models.get(source.datamodel.id, {}).items():
-            if subs == '*':  # either normal field or flow type (all blocks)
-                field = source[r_key]
-                if flatten and isinstance(field, Flow):
-                    for i, flow in enumerate(field.blocks):
-                        flowtype = flow['_flowblock']
-                        for f_key, block in flow._data.items():
-                            if f_key.startswith('_'):  # e.g., _flowblock
-                                continue
-                            yield FieldKeyPath(r_key, i, f_key), block
-                else:
-                    yield FieldKeyPath(r_key), field
-            else:  # always flow type (only some blocks)
-                for i, flow in enumerate(source[r_key].blocks):
-                    flowtype = flow['_flowblock']
-                    for f_key in _flows.get(flowtype, []):
-                        yield FieldKeyPath(r_key, i, f_key), flow[f_key]
-
-    def groupby(
-        self,
-        attrib: AttributeKey,
-        root: lektor.db.Record,
-        func: GroupingCallback,
-        flatten: bool = False,
-        incl_attachments: bool = True
-    ) -> Dict[GroupKey, List[GroupComponent]]:
-        '''
-        Traverse selected root record with all children and group by func.
-        Func is called with (record, FieldKeyPath, FieldValue).
-        Func may yield one or more (group-key, extra-info) tuples.
-
-        return {'group-key': [(record, extra-info), ...]}
-        '''
-        assert callable(func), 'no GroupingCallback provided'
-        assert isinstance(root, lektor.db.Record)
-        tmap = {}  # type: Dict[GroupKey, List[GroupComponent]]
-        recursive_list = [root]  # type: List[lektor.db.Record]
-        while recursive_list:
-            record = recursive_list.pop()
-            if hasattr(record, 'children'):
-                # recursive_list += record.children
-                recursive_list.extend(record.children)
-            if incl_attachments and hasattr(record, 'attachments'):
-                # recursive_list += record.attachments
-                recursive_list.extend(record.attachments)
-            for key, field in self.iter_record_fields(record, attrib, flatten):
-                for ret in func(GroupByCallbackArgs(record, key, field)) or []:
-                    assert isinstance(ret, (tuple, list)), \
-                        'Must return tuple (group-key, extra-info)'
-                    group_key, extras = ret
-                    if group_key not in tmap:
-                        tmap[group_key] = []
-                    tmap[group_key].append(GroupComponent(record, extras))
-        return tmap
-
-    # -----------------
-    #   Create groups
-    # -----------------
-
-    def should_process(self, node: SourceObject) -> bool:
-        ''' Check if record path is being watched. '''
-        return isinstance(node, lektor.db.Record) \
-            and node.url_path in self._func
-
-    def make_cluster(self, root: lektor.db.Record) -> Iterator[GroupBySource]:
-        ''' Group by attrib and build Artifacts. '''
-        assert isinstance(root, lektor.db.Record)
-        for attr, fn, fl, temp, slug, dep in self._func.get(root.url_path, []):
-            groups = self.groupby(attr, root, func=fn, flatten=fl)
-            for group_key, children in groups.items():
-                obj = GroupBySource(root, attr, group_key, children,
-                                    template=temp, slug=slug, dependency=dep)
-                self.track_dev_server_path(obj)
-                yield obj
-
-    # ------------------
-    #   Path resolving
-    # ------------------
-
-    def resolve_virtual_path(
-        self, node: SourceObject, pieces: List[str]
-    ) -> Optional[GroupBySource]:
-        ''' Given a @VPATH/attrib/groupkey path, determine url path. '''
-        if len(pieces) >= 2:
-            attrib: AttributeKey = pieces[0]  # type: ignore[assignment]
-            group: GroupKey = pieces[1]  # type: ignore[assignment]
-            for attr, _, _, _, slug, _ in self._func.get(node.url_path, []):
-                if attr == attrib:
-                    # TODO: do we need to provide the template too?
-                    return GroupBySource(node, attr, group, slug=slug)
-        return None
-
-    def track_dev_server_path(self, sender: GroupBySource) -> None:
-        ''' Dev server only: Add target path to reverse artifact url lookup '''
-        self._resolve_map[sender.url_path] = sender
-
-    def resolve_dev_server_path(
-        self, node: SourceObject, pieces: List[str]
-    ) -> Optional[GroupBySource]:
-        ''' Dev server only: Resolve actual url to virtual obj. '''
-        return self._resolve_map.get(build_url([node.url_path] + pieces))
-
-
-# -----------------------------------
-#           Plugin Entry
-# -----------------------------------
-
-
-class GroupByPlugin(Plugin):
-    name = 'GroupBy Plugin'
-    description = 'Cluster arbitrary records with field attribute keyword.'
-
-    def on_setup_env(self, **extra):
-        self.creator = GroupByCreator()
-        self.env.add_build_program(GroupBySource, GroupByBuildProgram)
-        # let other plugins register their @groupby.watch functions
-        self.emit('init', groupby=self.creator, **extra)
-
-        # resolve /tag/rss/ -> /tag/rss/index.html (local server only)
-        @self.env.urlresolver
-        def groupby_path_resolver(node, pieces):
-            if self.creator.should_process(node):
-                return self.creator.resolve_dev_server_path(node, pieces)
-
-        # use VPATH in templates: {{ '/@groupby/attrib/group' | url }}
-        @self.env.virtualpathresolver(VPATH.lstrip('@'))
-        def groupby_virtualpath_resolver(node, pieces):
-            if self.creator.should_process(node):
-                return self.creator.resolve_virtual_path(node, pieces)
-
-        # injection to generate GroupBy nodes when processing artifacts
-        # @self.env.generator
-        # def groupby_generator(node):
-        #     if self.creator.should_process(node):
-        #         yield from self.creator.make_cluster(node)
-
-    def _quick_config(self):
-        config = self.get_config()
-        for attrib in config.sections():
-            sect = config.section_as_dict(attrib)
-            root = sect.get('root', '/')
-            slug = sect.get('slug')
-            temp = sect.get('template')
-            split = sect.get('split')
-
-            @self.creator.watch_once(root, attrib, template=temp, slug=slug,
-                                     dependency=self.config_filename)
-            def _fn(args):
-                val = args.field
-                if isinstance(val, str):
-                    val = val.split(split) if split else [val]  # make list
-                if isinstance(val, list):
-                    for tag in val:
-                        yield slugify(tag), tag
-
-    def on_before_build_all(self, builder, **extra):
-        # let other plugins register their @groupby.watch_once functions
-        self.emit('init-once', groupby=self.creator, builder=builder, **extra)
-        # load config file quick listeners (before initialize!)
-        self._quick_config()
-        # parse all models to detect attribs of listeners
-        self.creator.initialize(builder.pad.db)
-
-    def on_before_build(self, builder, build_state, source, prog, **extra):
-        # Injection to create GroupBy nodes before parent node is built.
-        # Use this callback (not @generator) to modify parent beforehand.
-        # Relevant for the root page which is otherwise build before GroupBy.
-        if self.creator.should_process(source):
-            for vobj in self.creator.make_cluster(source):
-                builder.build(vobj)
-
-    def on_after_build_all(self, builder, **extra):
-        # remove all quick listeners (will be added again in the next build)
-        self.creator.remove_watch_once()
-
-    def on_after_prune(self, builder, **extra):
-        # TODO: find better way to prune unreferenced elements
-        GroupByPruner.prune(builder)

lektor_groupby/__init__.py

@@ -0,0 +1,4 @@
+from .config import Config  # noqa: F401
+from .groupby import GroupBy  # noqa: F401
+from .plugin import GroupByPlugin  # noqa: F401
+from .watcher import GroupByCallbackArgs  # noqa: F401

lektor_groupby/backref.py

@@ -0,0 +1,83 @@
+from lektor.context import get_ctx
+from typing import TYPE_CHECKING, Union, Iterable, Iterator
+import weakref
+if TYPE_CHECKING:
+    from lektor.builder import Builder
+    from lektor.db import Record
+    from .groupby import GroupBy
+    from .model import FieldKeyPath
+    from .vobj import GroupBySource
+
+
+class WeakVGroupsList(list):
+    def add(self, strong: 'FieldKeyPath', weak: 'GroupBySource') -> None:
+        super().append((strong, weakref.ref(weak)))
+
+
+class GroupByRef:
+    @staticmethod
+    def of(builder: 'Builder') -> 'GroupBy':
+        ''' Get the GroupBy object of a builder. '''
+        return builder.__groupby  # type:ignore[attr-defined,no-any-return]
+
+    @staticmethod
+    def set(builder: 'Builder', groupby: 'GroupBy') -> None:
+        ''' Set the GroupBy object of a builder. '''
+        builder.__groupby = groupby  # type: ignore[attr-defined]
+
+
+class VGroups:
+    @staticmethod
+    def of(record: 'Record') -> WeakVGroupsList:
+        '''
+        Return the (weak) set of virtual objects of a page.
+        Creates a new set if it does not exist yet.
+        '''
+        try:
+            wset = record.__vgroups  # type: ignore[attr-defined]
+        except AttributeError:
+            wset = WeakVGroupsList()
+            record.__vgroups = wset  # type: ignore[attr-defined]
+        return wset  # type: ignore[no-any-return]
+
+    @staticmethod
+    def iter(
+        record: 'Record',
+        keys: Union[str, Iterable[str], None] = None,
+        *,
+        fields: Union[str, Iterable[str], None] = None,
+        flows: Union[str, Iterable[str], None] = None,
+        recursive: bool = False
+    ) -> Iterator['GroupBySource']:
+        ''' Extract all referencing groupby virtual objects from a page. '''
+        ctx = get_ctx()
+        if not ctx:
+            raise NotImplementedError("Shouldn't happen, where is my context?")
+        # get GroupBy object
+        builder = ctx.build_state.builder
+        groupby = GroupByRef.of(builder)
+        groupby.make_once(builder)  # ensure did cluster before
+        # manage config dependencies
+        for dep in groupby.dependencies:
+            ctx.record_dependency(dep)
+        # prepare filter
+        if isinstance(keys, str):
+            keys = [keys]
+        if isinstance(fields, str):
+            fields = [fields]
+        if isinstance(flows, str):
+            flows = [flows]
+        # find groups
+        proc_list = [record]
+        while proc_list:
+            page = proc_list.pop(0)
+            if recursive and hasattr(page, 'children'):
+                proc_list.extend(page.children)  # type: ignore[attr-defined]
+            for key, vobj in VGroups.of(page):
+                if fields and key.fieldKey not in fields:
+                    continue
+                if flows and key.flowKey not in flows:
+                    continue
+                if keys and vobj().config.key not in keys:
+                    continue
+                yield vobj()

lektor_groupby/config.py

@@ -0,0 +1,87 @@
+from inifile import IniFile
+from lektor.utils import slugify
+
+from typing import Set, Dict, Optional, Union, Any
+
+AnyConfig = Union['Config', IniFile, Dict]
+
+
+class Config:
+    '''
+    Holds information for GroupByWatcher and GroupBySource.
+    This object is accessible in your template file ({{this.config}}).
+
+    Available attributes:
+    key, root, slug, template, enabled, dependencies, fields, key_map
+    '''
+
+    def __init__(
+        self,
+        key: str, *,
+        root: Optional[str] = None,  # default: "/"
+        slug: Optional[str] = None,  # default: "{attr}/{group}/index.html"
+        template: Optional[str] = None,  # default: "groupby-{attr}.html"
+    ) -> None:
+        self.key = key
+        self.root = (root or '/').rstrip('/') or '/'
+        self.slug = slug or (key + '/{key}/')  # key = GroupBySource.key
+        self.template = template or f'groupby-{self.key}.html'
+        # editable after init
+        self.enabled = True
+        self.dependencies = set()  # type: Set[str]
+        self.fields = {}  # type: Dict[str, Any]
+        self.key_map = {}  # type: Dict[str, str]
+
+    def slugify(self, k: str) -> str:
+        ''' key_map replace and slugify. '''
+        return slugify(self.key_map.get(k, k))  # type: ignore[no-any-return]
+
+    def set_fields(self, fields: Optional[Dict[str, Any]]) -> None:
+        '''
+        The fields dict is a mapping of attrib = Expression values.
+        Each dict key will be added to the GroupBySource virtual object.
+        Each dict value is passed through jinja context first.
+        '''
+        self.fields = fields or {}
+
+    def set_key_map(self, key_map: Optional[Dict[str, str]]) -> None:
+        ''' This mapping replaces group keys before slugify. '''
+        self.key_map = key_map or {}
+
+    def __repr__(self) -> str:
+        txt = '<GroupByConfig'
+        for x in ['key', 'root', 'slug', 'template', 'enabled']:
+            txt += ' {}="{}"'.format(x, getattr(self, x))
+        txt += f' fields="{", ".join(self.fields)}"'
+        return txt + '>'
+
+    @staticmethod
+    def from_dict(key: str, cfg: Dict[str, str]) -> 'Config':
+        ''' Set config fields manually. Allowed: key, root, slug, template. '''
+        return Config(
+            key=key,
+            root=cfg.get('root'),
+            slug=cfg.get('slug'),
+            template=cfg.get('template'),
+        )
+
+    @staticmethod
+    def from_ini(key: str, ini: IniFile) -> 'Config':
+        ''' Read and parse ini file. Also adds dependency tracking. '''
+        cfg = ini.section_as_dict(key)  # type: Dict[str, str]
+        conf = Config.from_dict(key, cfg)
+        conf.enabled = ini.get_bool(key + '.enabled', True)
+        conf.dependencies.add(ini.filename)
+        conf.set_fields(ini.section_as_dict(key + '.fields'))
+        conf.set_key_map(ini.section_as_dict(key + '.key_map'))
+        return conf
+
+    @staticmethod
+    def from_any(key: str, config: AnyConfig) -> 'Config':
+        assert isinstance(config, (Config, IniFile, Dict))
+        if isinstance(config, Config):
+            return config
+        elif isinstance(config, IniFile):
+            return Config.from_ini(key, config)
+        elif isinstance(config, Dict):
+            return Config.from_dict(key, config)

lektor_groupby/groupby.py

@@ -0,0 +1,80 @@
+from lektor.builder import PathCache
+from lektor.db import Record  # isinstance
+from typing import TYPE_CHECKING, Set, List
+from .config import Config
+from .watcher import Watcher
+if TYPE_CHECKING:
+    from .config import AnyConfig
+    from lektor.builder import Builder
+    from lektor.sourceobj import SourceObject
+    from .resolver import Resolver
+    from .vobj import GroupBySource
+
+
+class GroupBy:
+    '''
+    Process all children with matching conditions under specified page.
+    Creates a grouping of pages with similar (self-defined) attributes.
+    The grouping is performed only once per build.
+    '''
+
+    def __init__(self, resolver: 'Resolver') -> None:
+        self._watcher = []  # type: List[Watcher]
+        self._results = []  # type: List[GroupBySource]
+        self.resolver = resolver
+
+    def add_watcher(self, key: str, config: 'AnyConfig') -> Watcher:
+        ''' Init Config and add to watch list. '''
+        w = Watcher(Config.from_any(key, config))
+        self._watcher.append(w)
+        return w
+
+    def get_dependencies(self) -> Set[str]:
+        deps = set()  # type: Set[str]
+        for w in self._watcher:
+            deps.update(w.config.dependencies)
+        return deps
+
+    def queue_all(self, builder: 'Builder') -> None:
+        ''' Iterate full site-tree and queue all children. '''
+        self.dependencies = self.get_dependencies()
+        # remove disabled watchers
+        self._watcher = [w for w in self._watcher if w.config.enabled]
+        if not self._watcher:
+            return
+        # initialize remaining (enabled) watchers
+        for w in self._watcher:
+            w.initialize(builder.pad)
+        # iterate over whole build tree
+        queue = builder.pad.get_all_roots()  # type: List[SourceObject]
+        while queue:
+            record = queue.pop()
+            if hasattr(record, 'attachments'):
+                queue.extend(record.attachments)  # type: ignore[attr-defined]
+            if hasattr(record, 'children'):
+                queue.extend(record.children)  # type: ignore[attr-defined]
+            if isinstance(record, Record):
+                for w in self._watcher:
+                    if w.should_process(record):
+                        w.process(record)
+
+    def make_once(self, builder: 'Builder') -> None:
+        ''' Perform groupby, iter over sources with watcher callback. '''
+        if self._watcher:
+            self.resolver.reset()
+            for w in self._watcher:
+                root = builder.pad.get(w.config.root)
+                for vobj in w.iter_sources(root):
+                    self._results.append(vobj)
+                    self.resolver.add(vobj)
+            self._watcher.clear()
+
+    def build_all(self, builder: 'Builder') -> None:
+        ''' Create virtual objects and build sources. '''
+        self.make_once(builder)  # in case no page used the |vgroups filter
+        path_cache = PathCache(builder.env)
+        for vobj in self._results:
+            if vobj.slug:
+                builder.build(vobj, path_cache)
+        del path_cache
+        self._results.clear()  # garbage collect weak refs

lektor_groupby/model.py

@@ -0,0 +1,66 @@
+from lektor.db import Database, Record  # typing
+from lektor.types.flow import Flow, FlowType
+from lektor.utils import bool_from_string
+from typing import Set, Dict, Tuple, Any, NamedTuple, Optional, Iterator
+
+
+class FieldKeyPath(NamedTuple):
+    fieldKey: str
+    flowIndex: Optional[int] = None
+    flowKey: Optional[str] = None
+
+
+class ModelReader:
+    '''
+    Find models and flow-models which contain attribute.
+    Flows are either returned directly (flatten=False) or
+    expanded so that each flow-block is yielded (flatten=True)
+    '''
+
+    def __init__(self, db: Database, attr: str, flatten: bool = False) -> None:
+        self.flatten = flatten
+        self._flows = {}  # type: Dict[str, Set[str]]
+        self._models = {}  # type: Dict[str, Dict[str, str]]
+        # find flow blocks containing attribute
+        for key, flow in db.flowblocks.items():
+            tmp1 = set(f.name for f in flow.fields
+                       if bool_from_string(f.options.get(attr, False)))
+            if tmp1:
+                self._flows[key] = tmp1
+        # find models and flow-blocks containing attribute
+        for key, model in db.datamodels.items():
+            tmp2 = {}  # Dict[str, str]
+            for field in model.fields:
+                if bool_from_string(field.options.get(attr, False)):
+                    tmp2[field.name] = '*'  # include all children
+                elif isinstance(field.type, FlowType) and self._flows:
+                    # only processed if at least one flow has attr
+                    fbs = field.type.flow_blocks
+                    # if fbs == None, all flow-blocks are allowed
+                    if fbs is None or any(x in self._flows for x in fbs):
+                        tmp2[field.name] = '?'  # only some flow blocks
+            if tmp2:
+                self._models[key] = tmp2
+
+    def read(self, record: Record) -> Iterator[Tuple[FieldKeyPath, Any]]:
+        ''' Enumerate all fields of a Record with attrib = True. '''
+        assert isinstance(record, Record)
+        for r_key, subs in self._models.get(record.datamodel.id, {}).items():
+            field = record[r_key]
+            if not field:
+                continue
+            if subs == '*':  # either normal field or flow type (all blocks)
+                if self.flatten and isinstance(field, Flow):
+                    for i, flow in enumerate(field.blocks):
+                        flowtype = flow['_flowblock']
+                        for f_key, block in flow._data.items():
+                            if f_key.startswith('_'):  # e.g., _flowblock
+                                continue
+                            yield FieldKeyPath(r_key, i, f_key), block
+                else:
+                    yield FieldKeyPath(r_key), field
+            else:  # always flow type (only some blocks)
+                for i, flow in enumerate(field.blocks):
+                    flowtype = flow['_flowblock']
+                    for f_key in self._flows.get(flowtype, []):
+                        yield FieldKeyPath(r_key, i, f_key), flow[f_key]

lektor_groupby/plugin.py

@@ -0,0 +1,81 @@
+from lektor.db import Page  # isinstance
+from lektor.pluginsystem import Plugin  # subclass
+from typing import TYPE_CHECKING, Iterator, Any
+from .backref import GroupByRef, VGroups
+from .groupby import GroupBy
+from .pruner import prune
+from .resolver import Resolver
+from .vobj import VPATH, GroupBySource, GroupByBuildProgram
+if TYPE_CHECKING:
+    from lektor.builder import Builder, BuildState
+    from lektor.sourceobj import SourceObject
+    from .watcher import GroupByCallbackArgs
+
+
+class GroupByPlugin(Plugin):
+    name = 'GroupBy Plugin'
+    description = 'Cluster arbitrary records with field attribute keyword.'
+
+    def on_setup_env(self, **extra: Any) -> None:
+        self.has_changes = False
+        self.resolver = Resolver(self.env)
+        self.env.add_build_program(GroupBySource, GroupByBuildProgram)
+        self.env.jinja_env.filters.update(vgroups=VGroups.iter)
+
+    def on_before_build(
+        self, builder: 'Builder', source: 'SourceObject', **extra: Any
+    ) -> None:
+        # before-build may be called before before-build-all (issue #1017)
+        # make sure it is always evaluated first
+        if isinstance(source, Page):
+            self._init_once(builder)
+
+    def on_after_build(self, build_state: 'BuildState', **extra: Any) -> None:
+        if build_state.updated_artifacts:
+            self.has_changes = True
+
+    def on_after_build_all(self, builder: 'Builder', **extra: Any) -> None:
+        # only rebuild if has changes (bypass idle builds)
+        # or the very first time after startup (url resolver & pruning)
+        if self.has_changes or not self.resolver.has_any:
+            self._init_once(builder).build_all(builder)  # updates resolver
+            self.has_changes = False
+
+    def on_after_prune(self, builder: 'Builder', **extra: Any) -> None:
+        # TODO: find a better way to prune unreferenced elements
+        prune(builder, VPATH, self.resolver.files)
+
+    # ------------
+    #   internal
+    # ------------
+
+    def _init_once(self, builder: 'Builder') -> GroupBy:
+        try:
+            return GroupByRef.of(builder)
+        except AttributeError:
+            groupby = GroupBy(self.resolver)
+            GroupByRef.set(builder, groupby)
+
+        self._load_quick_config(groupby)
+        # let other plugins register their @groupby.watch functions
+        self.emit('before-build-all', groupby=groupby, builder=builder)
+        groupby.queue_all(builder)
+        return groupby
+
+    def _load_quick_config(self, groupby: GroupBy) -> None:
+        ''' Load config file quick listeners. '''
+        config = self.get_config()
+        for key in config.sections():
+            if '.' in key:  # e.g., key.fields and key.key_map
+                continue
+
+            watcher = groupby.add_watcher(key, config)
+            split = config.get(key + '.split')  # type: str
+
+            @watcher.grouping()
+            def _fn(args: 'GroupByCallbackArgs') -> Iterator[str]:
+                val = args.field
+                if isinstance(val, str):
+                    val = map(str.strip, val.split(split)) if split else [val]
+                if isinstance(val, (list, map)):
+                    yield from val

lektor_groupby/pruner.py

@@ -0,0 +1,45 @@
+'''
+Static collector for build-artifact urls.
+All non-tracked VPATH-urls will be pruned after build.
+'''
+from lektor.reporter import reporter  # report_pruned_artifact
+from lektor.utils import prune_file_and_folder
+from typing import TYPE_CHECKING, Set, Iterable
+if TYPE_CHECKING:
+    from lektor.builder import Builder
+
+
+def _normalize_url_cache(url_cache: Iterable[str]) -> Set[str]:
+    cache = set()
+    for url in url_cache:
+        if url.endswith('/'):
+            url += 'index.html'
+        cache.add(url.lstrip('/'))
+    return cache
+
+
+def prune(builder: 'Builder', vpath: str, url_cache: Iterable[str]) -> None:
+    '''
+    Remove previously generated, unreferenced Artifacts.
+    All urls in url_cache must have a trailing "/index.html" (instead of "/")
+    and also, no leading slash, "blog/index.html" instead of "/blog/index.html"
+    '''
+    vpath = '@' + vpath.lstrip('@')  # just in case of user error
+    dest_path = builder.destination_path
+    url_cache = _normalize_url_cache(url_cache)
+    con = builder.connect_to_database()
+    try:
+        with builder.new_build_state() as build_state:
+            for url, file in build_state.iter_artifacts():
+                if url.lstrip('/') in url_cache:
+                    continue  # generated in this build-run
+                infos = build_state.get_artifact_dependency_infos(url, [])
+                for artifact_name, _ in infos:
+                    if vpath not in artifact_name:
+                        continue  # we only care about our Virtuals
+                    reporter.report_pruned_artifact(url)
+                    prune_file_and_folder(file.filename, dest_path)
+                    build_state.remove_artifact(url)
+                    break  # there is only one VPATH-entry per source
+    finally:
+        con.close()

lektor_groupby/resolver.py

@@ -0,0 +1,61 @@
+from lektor.db import Record  # isinstance
+from lektor.utils import build_url
+from typing import TYPE_CHECKING, Dict, List, Tuple, Optional, Iterable
+from .vobj import VPATH, GroupBySource
+if TYPE_CHECKING:
+    from lektor.environment import Environment
+    from lektor.sourceobj import SourceObject
+    from .config import Config
+
+
+class Resolver:
+    '''
+    Resolve virtual paths and urls ending in /.
+    Init will subscribe to @urlresolver and @virtualpathresolver.
+    '''
+
+    def __init__(self, env: 'Environment') -> None:
+        self._data = {}  # type: Dict[str, Tuple[str, str, Config]]
+        env.urlresolver(self.resolve_server_path)
+        env.virtualpathresolver(VPATH.lstrip('@'))(self.resolve_virtual_path)
+
+    @property
+    def has_any(self) -> bool:
+        return bool(self._data)
+
+    @property
+    def files(self) -> Iterable[str]:
+        return self._data
+
+    def reset(self) -> None:
+        ''' Clear previously recorded virtual objects. '''
+        self._data.clear()
+
+    def add(self, vobj: GroupBySource) -> None:
+        ''' Track new virtual object (only if slug is set). '''
+        if vobj.slug:
+            self._data[vobj.url_path] = (vobj.key, vobj.group, vobj.config)
+
+    # ------------
+    #   Resolver
+    # ------------
+
+    def resolve_server_path(self, node: 'SourceObject', pieces: List[str]) \
+            -> Optional[GroupBySource]:
+        ''' Local server only: resolve /tag/rss/ -> /tag/rss/index.html '''
+        if isinstance(node, Record):
+            rv = self._data.get(build_url([node.url_path] + pieces))
+            if rv:
+                return GroupBySource(node, rv[0]).finalize(rv[2], rv[1])
+        return None
+
+    def resolve_virtual_path(self, node: 'SourceObject', pieces: List[str]) \
+            -> Optional[GroupBySource]:
+        ''' Admin UI only: Prevent server error and null-redirect. '''
+        if isinstance(node, Record) and len(pieces) >= 2:
+            path = node['_path']  # type: str
+            attr, grp, *_ = pieces
+            for slug, group, conf in self._data.values():
+                if attr == conf.key and slug == grp and path == conf.root:
+                    return GroupBySource(node, slug).finalize(conf, group)
+        return None

lektor_groupby/util.py

@@ -0,0 +1,28 @@
+from lektor.reporter import reporter, style
+
+from typing import List, Dict
+
+
+def report_config_error(key: str, field: str, val: str, e: Exception) -> None:
+    ''' Send error message to Lektor reporter. Indicate which field is bad. '''
+    msg = '[ERROR] invalid config for [{}.{}] = "{}",  Error: {}'.format(
+        key, field, val, repr(e))
+    try:
+        reporter._write_line(style(msg, fg='red'))
+    except Exception:
+        print(msg)  # fallback in case Lektor API changes
+
+
+def most_used_key(keys: List[str]) -> str:
+    if len(keys) < 3:
+        return keys[0]  # TODO: first vs last occurrence
+    best_count = 0
+    best_key = ''
+    tmp = {}  # type: Dict[str, int]
+    for k in keys:
+        num = (tmp[k] + 1) if k in tmp else 1
+        tmp[k] = num
+        if num > best_count:  # TODO: (>) vs (>=), first vs last occurrence
+            best_count = num
+            best_key = k
+    return best_key

lektor_groupby/vobj.py

@@ -0,0 +1,163 @@
+from lektor.build_programs import BuildProgram  # subclass
+from lektor.context import get_ctx
+from lektor.environment import Expression
+from lektor.sourceobj import VirtualSourceObject  # subclass
+from lektor.utils import build_url
+from typing import TYPE_CHECKING, Dict, List, Any, Optional, Iterator
+from .util import report_config_error, most_used_key
+if TYPE_CHECKING:
+    from lektor.builder import Artifact
+    from lektor.db import Record
+    from .config import Config
+
+VPATH = '@groupby'  # potentially unsafe. All matching entries are pruned.
+
+
+# -----------------------------------
+#           VirtualSource
+# -----------------------------------
+
+class GroupBySource(VirtualSourceObject):
+    '''
+    Holds information for a single group/cluster.
+    This object is accessible in your template file.
+    Attributes: record, key, group, slug, children, config
+    '''
+
+    def __init__(self, record: 'Record', slug: str) -> None:
+        super().__init__(record)
+        self.key = slug
+        self._group_map = []  # type: List[str]
+        self._children = {}  # type: Dict[Record, List[Any]]
+
+    def append_child(self, child: 'Record', extra: Any, group: str) -> None:
+        if child not in self._children:
+            self._children[child] = [extra]
+        else:
+            self._children[child].append(extra)
+        # _group_map is later used to find most used group
+        self._group_map.append(group)
+
+    # -------------------------
+    #   Evaluate Extra Fields
+    # -------------------------
+
+    def finalize(self, config: 'Config', group: Optional[str] = None) \
+            -> 'GroupBySource':
+        self.config = config
+        self.group = group or most_used_key(self._group_map)
+        del self._group_map
+        # evaluate slug Expression
+        if config.slug and '{key}' in config.slug:
+            self.slug = config.slug.replace('{key}', self.key)
+        else:
+            self.slug = self._eval(config.slug, field='slug')
+            assert self.slug != Ellipsis, 'invalid config: ' + config.slug
+        if self.slug and self.slug.endswith('/index.html'):
+            self.slug = self.slug[:-10]
+        # extra fields
+        for attr, expr in config.fields.items():
+            setattr(self, attr, self._eval(expr, field='fields.' + attr))
+        return self
+
+    def _eval(self, value: Any, *, field: str) -> Any:
+        ''' Internal only: evaluates Lektor config file field expression. '''
+        if not isinstance(value, str):
+            return value
+        pad = self.record.pad
+        alt = self.record.alt
+        try:
+            return Expression(pad.env, value).evaluate(pad, this=self, alt=alt)
+        except Exception as e:
+            report_config_error(self.config.key, field, value, e)
+            return Ellipsis
+
+    # ---------------------
+    #   Lektor properties
+    # ---------------------
+
+    @property
+    def path(self) -> str:
+        # Used in VirtualSourceInfo, used to prune VirtualObjects
+        return f'{self.record.path}{VPATH}/{self.config.key}/{self.key}'
+
+    @property
+    def url_path(self) -> str:
+        # Actual path to resource as seen by the browser
+        return build_url([self.record.path, self.slug])  # slug can be None!
+
+    def iter_source_filenames(self) -> Iterator[str]:
+        ''' Enumerate all dependencies '''
+        if self.config.dependencies:
+            yield from self.config.dependencies
+        for record in self._children:
+            yield from record.iter_source_filenames()
+
+    # -----------------------
+    #   Properties & Helper
+    # -----------------------
+
+    @property
+    def children(self) -> Dict['Record', List[Any]]:
+        ''' Returns dict with page record key and (optional) extra value. '''
+        return self._children
+
+    @property
+    def first_child(self) -> Optional['Record']:
+        ''' Returns first referencing page record. '''
+        if self._children:
+            return iter(self._children).__next__()
+        return None
+
+    @property
+    def first_extra(self) -> Optional[Any]:
+        ''' Returns first additional / extra info object of first page. '''
+        if not self._children:
+            return None
+        val = iter(self._children.values()).__next__()
+        return val[0] if val else None
+
+    def __getitem__(self, key: str) -> Any:
+        # Used for virtual path resolver
+        if key in ('_path', '_alt'):
+            return getattr(self, key[1:])
+        return self.__missing__(key)  # type: ignore[attr-defined]
+
+    def __lt__(self, other: 'GroupBySource') -> bool:
+        # Used for |sort filter ("group" is the provided original string)
+        return self.group.lower() < other.group.lower()
+
+    def __eq__(self, other: object) -> bool:
+        # Used for |unique filter
+        if self is other:
+            return True
+        return isinstance(other, GroupBySource) and \
+            self.path == other.path and self.slug == other.slug
+
+    def __hash__(self) -> int:
+        # Used for hashing in set and dict
+        return hash((self.path, self.slug))
+
+    def __repr__(self) -> str:
+        return '<GroupBySource path="{}" children={}>'.format(
+            self.path, len(self._children))
+
+
+# -----------------------------------
+#           BuildProgram
+# -----------------------------------
+
+class GroupByBuildProgram(BuildProgram):
+    ''' Generate Build-Artifacts and write files. '''
+
+    def produce_artifacts(self) -> None:
+        url = self.source.url_path
+        if url.endswith('/'):
+            url += 'index.html'
+        self.declare_artifact(url, sources=list(
+            self.source.iter_source_filenames()))
+
+    def build_artifact(self, artifact: 'Artifact') -> None:
+        get_ctx().record_virtual_dependency(self.source)
+        artifact.render_template_into(
+            self.source.config.template, this=self.source)

lektor_groupby/watcher.py

@@ -0,0 +1,111 @@
+from typing import TYPE_CHECKING, Dict, List, Tuple, Any, Union, NamedTuple
+from typing import Optional, Callable, Iterator, Generator
+from .backref import VGroups
+from .model import ModelReader
+from .vobj import GroupBySource
+if TYPE_CHECKING:
+    from lektor.db import Pad, Record
+    from .config import Config
+    from .model import FieldKeyPath
+
+
+class GroupByCallbackArgs(NamedTuple):
+    record: 'Record'
+    key: 'FieldKeyPath'
+    field: Any  # lektor model data-field value
+
+
+GroupingCallback = Callable[[GroupByCallbackArgs], Union[
+    Iterator[Union[str, Tuple[str, Any]]],
+    Generator[Union[str, Tuple[str, Any]], Optional[str], None],
+]]
+
+
+class Watcher:
+    '''
+    Callback is called with (Record, FieldKeyPath, field-value).
+    Callback may yield one or more (group, extra-info) tuples.
+    '''
+
+    def __init__(self, config: 'Config') -> None:
+        self.config = config
+        self._root = self.config.root
+
+    def grouping(self, flatten: bool = True) \
+            -> Callable[[GroupingCallback], None]:
+        '''
+        Decorator to subscribe to attrib-elements.
+        If flatten = False, dont explode FlowType.
+
+        (record, field-key, field) -> (group, extra-info)
+        '''
+        def _decorator(fn: GroupingCallback) -> None:
+            self.flatten = flatten
+            self.callback = fn
+        return _decorator
+
+    def initialize(self, pad: 'Pad') -> None:
+        ''' Reset internal state. You must initialize before each build! '''
+        assert callable(self.callback), 'No grouping callback provided.'
+        self._model_reader = ModelReader(pad.db, self.config.key, self.flatten)
+        self._root_record = pad.get(self._root)  # type: Record
+        self._state = {}  # type: Dict[str, GroupBySource]
+
+    def should_process(self, node: 'Record') -> bool:
+        ''' Check if record path is being watched. '''
+        return node['_path'].startswith(self._root)
+
+    def process(self, record: 'Record') -> None:
+        '''
+        Will iterate over all record fields and call the callback method.
+        Each record is guaranteed to be processed only once.
+        '''
+        for key, field in self._model_reader.read(record):
+            _gen = self.callback(GroupByCallbackArgs(record, key, field))
+            try:
+                obj = next(_gen)
+                while True:
+                    if not isinstance(obj, (str, tuple)):
+                        raise TypeError(f'Unsupported groupby yield: {obj}')
+                    slug = self._persist(record, key, obj)
+                    # return slugified group key and continue iteration
+                    if isinstance(_gen, Generator) and not _gen.gi_yieldfrom:
+                        obj = _gen.send(slug)
+                    else:
+                        obj = next(_gen)
+            except StopIteration:
+                del _gen
+
+    def _persist(
+        self, record: 'Record', key: 'FieldKeyPath', obj: Union[str, tuple]
+    ) -> str:
+        ''' Update internal state. Return slugified string. '''
+        if isinstance(obj, str):
+            group, extra = obj, key.fieldKey
+        else:
+            group, extra = obj
+
+        slug = self.config.slugify(group)
+        if slug not in self._state:
+            src = GroupBySource(self._root_record, slug)
+            self._state[slug] = src
+        else:
+            src = self._state[slug]
+
+        src.append_child(record, extra, group)
+        # reverse reference
+        VGroups.of(record).add(key, src)
+        return slug
+
+    def iter_sources(self, root: 'Record') -> Iterator[GroupBySource]:
+        ''' Prepare and yield GroupBySource elements. '''
+        for vobj in self._state.values():
+            yield vobj.finalize(self.config)
+        # cleanup. remove this code if you'd like to iter twice
+        del self._model_reader
+        del self._root_record
+        del self._state
+
+    def __repr__(self) -> str:
+        return '<GroupByWatcher key="{}" enabled={} callback={}>'.format(
+            self.config.key, self.config.enabled, self.callback)

setup.py

@@ -1,15 +1,11 @@
 from setuptools import setup
-import re
 
 with open('README.md') as fp:
     longdesc = fp.read()
-    # replace fragment links with bold text
-    frag_links = re.compile(r'\[([^]]+)\]\(#[^)]*\)')
-    longdesc = frag_links.sub(lambda x: '__{}__'.format(x.group(1)), longdesc)
 
 setup(
     name='lektor-groupby',
-    py_modules=['lektor_groupby'],
+    packages=['lektor_groupby'],
     entry_points={
         'lektor.plugins': [
             'groupby = lektor_groupby:GroupByPlugin',
@@ -17,7 +13,7 @@ setup(
     },
     author='relikd',
     url='https://github.com/relikd/lektor-groupby-plugin',
-    version='0.9',
+    version='0.9.7',
     description='Cluster arbitrary records with field attribute keyword.',
     long_description=longdesc,
     long_description_content_type="text/markdown",
@@ -31,7 +27,6 @@ setup(
         'cluster',
     ],
     classifiers=[
-        'Development Status :: 5 - Production/Stable',
         'Environment :: Web Environment',
         'Environment :: Plugins',
         'Framework :: Lektor',