v0.9.3

update example readme v0.9.3
builtin filter collision rename groupby -> vgroups
2022-04-06 15:47:38 +02:00 · 2022-04-06 15:42:02 +02:00 · 2022-04-06 13:29:19 +02:00 · 2022-04-06 13:16:44 +02:00 · 2022-04-06 13:11:49 +02:00 · 2022-04-06 12:29:35 +02:00
34 changed files with 1281 additions and 640 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1 @@
 examples/** linguist-documentation
--- a/README.md
+++ b/README.md
@@ -1,168 +1,26 @@
 # Lektor Plugin: groupby
-A generic grouping / clustering plugin. Can be used for tagging or similar tasks.
+A generic grouping / clustering plugin.
 Can be used for tagging or similar tasks.
 The grouping algorithm is performed once.
 Contrary to, at least, cubic runtime if doing the same with Pad queries.
-Overview:
+Install this plugin or modify your Lektor project file:
 - the [basic example](#usage-basic-example) goes into detail how this plugin works.
 - the [quick config](#usage-quick-config) example show how you can use the plugin config to setup a quick and easy tagging system.
 - the [complex example](#usage-a-slightly-more-complex-example) touches on the potential of what is possible.
 ```sh
 lektor plugin add groupby
 ```
-## Usage: Basic example
+Optionally, enable a basic config:
 Lets start with a simple example: adding a tags field to your model.
 Assuming you have a `blog-entry.ini` that is used for all children of `/blog` path.
 #### `models/blog-entry.ini`
 ```ini
-[fields.tags]
+[tags]
-label = Tags
+root = /
-type = strings
+slug = tag/{group}.html
-myvar = true
+template = tag.html
 [fields.body]
 label = Content
 type = markdown
 ```
 Notice we introduce a new attribute variable: `myvar = true`.
 The name can be anything here, we will come to that later.
 The only thing that matters is that the value is a boolean and set to true.
 Edit your blog entry and add these two new tags:
 ```
 Awesome
 Latest News
 ```
 Next, we need a plugin to add the groupby event listener.
 #### `packages/test/lektor_my_tags_plugin.py`
 ```python
 def on_groupby_init(self, groupby, **extra):
    @groupby.watch('/blog', 'myvar', flatten=True, template='myvar.html',
                   slug='tag/{group}/index.html')
    def do_myvar(args):
        page = args.record  # extract additional info from source
        fieldKey, flowIndex, flowKey = args.key  # or get field index directly
        # val = page.get(fieldKey).blocks[flowIndex].get(flowKey)
        value = args.field  # list type since model is 'strings' type
        for tag in value:
            yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
 ```
 There are a few important things here:
 1. The first parameter (`'/blog'`) is the root page of the groupby.
   All results will be placed under this directory, e.g., `/blog/tags/clean/`.
   You can also just use `/`, in which case the same path would be `/tags/clean/`.
   Or create multiple listeners, one for `/blog/` and another for `/projects/`, etc.
 2. The second parameter (`'myvar'`) must be the same attribute variable we used in our `blog-entry.ini` model.
   The groupby plugin will traverse all models and search for this attribute name.
 3. Flatten determines how Flow elements are processed.
   If `False`, the callback function `convert_myvar()` is called once per Flow element (if the Flow element has the `myvar` attribute attached).
   If `True` (default), the callback is called for all Flow blocks individually.
 4. The template `myvar.html` is used to render the grouping page.
   This parameter is optional.
   If no explicit template is set, the default template `groupby-myvar.html` would be used. Where `myvar` is replaced with whatever attribute you chose.
 5. Finally, the slug `tag/{group}/index.html` is where the result is placed.
   The default value for this parameter is `{attrib}/{group}/index.html`.
   In our case, the default path would resolve to `myvar/awesome/index.html`.
   We explicitly chose to replace the default slug with our own, which ignores the attrib path component and instead puts the result pages inside the `/tag` directory.
   (PS: you could also use for example `t/{group}.html`, etc.)
 So much for the `args` parameter.
 The callback body **can** produce groupings but does not have to.
 If you choose to produce an entry, you have to `yield` a tuple pair of `(groupkey, extra-info)`.
 `groupkey` is used to combine & cluster pages and must be URL-safe.
 The `extra-info` is passed through to your template file.
 You can yield more than one entry per source or filter / ignore pages if you don't yield anything.
 Our simple example will generate the output files `tag/awesome/index.html` and `tag/latest-news/index.html`.
 Lets take a look at the html next.
 #### `templates/myvar.html`
 ```html
 <h2>Path: {{ this | url(absolute=True) }}</h2>
 <div>This is: {{this}}</div>
 <ul>
 	{%- for child in this.children %}
 	<li>Page: {{ child.record.path }}, Name: {{ child.extra.val }}, Tag count: {{ child.extra.tags_in_page }}</li>
 	{%- endfor %}
 </ul>
 ```
 Notice, we can use `child.record` to access the referenced page of the group cluster.
 `child.extra` contains the additional information we previously passed into the template.
 The final result of `tag/latest-news/index.html`:
 ```
 Path: /tag/latest-news/
 This is: <GroupBySource attribute="myvar" group="latest-news" template="myvar.html" slug="tag/latest-news/" children=1>
  -  Page: /blog/barss, Name: Latest News, Tag count: 2
 ```
 ## Usage: Quick config
 The whole example above can be simplified with a plugin config:
 #### `configs/groupby.ini`
 ```ini
 [myvar]
 root = /blog/
 slug = tag/{group}/index.html
 template = myvar.html
 split = ' '
 ```
-You still need to add a separate attribute to your model (step 1), but anything else is handled by the config file.
+Or dive into plugin development...
 All of these fields are optional and fallback to the default values stated above.
-The newly introduced option `split` will be used as string delimiter.
+For usage examples, refer to the [examples](https://github.com/relikd/lektor-groupby-plugin/tree/main/examples) readme.
 This allows to have a field with `string` type instead of `strings` type.
 If you do not provide the `split` option, the whole field value will be used as group key.
 Note: split is only used on str fields (`string` type), not lists (`strings` type).
 The emitted `extra-info` for the child is the original key value.
 E.g., `Latest News,Awesome` with `split = ,` yields `('latest-news', 'Latest News')` and `('awesome', 'Awesome')`.
 ## Usage: A slightly more complex example
 There are situations though, where a simple config file is not enough.
 The following plugin will find all model fields with attribute `inlinetags` and search for in-text occurrences of `{{Tagname}}` etc.
 ```python
 from lektor.markdown import Markdown
 from lektor.types.formats import MarkdownDescriptor
 from lektor.utils import slugify
 import re
 _regex = re.compile(r'{{([^}]{1,32})}}')
 def on_groupby_init(self, groupby, **extra):
    @groupby.watch('/', 'inlinetags', slug='tags/{group}/')
    def convert_inlinetags(args):
        arr = args.field if isinstance(args.field, list) else [args.field]
        for obj in arr:
            if isinstance(obj, (Markdown, MarkdownDescriptor)):
                obj = obj.source
            if isinstance(obj, str) and str:
                for match in _regex.finditer(obj):
                    tag = match.group(1)
                    yield slugify(tag), tag
 ```
 This generic approach does not care what data-type the field value is:
 `strings` fields will be expanded and enumerated, Markdown will be unpacked.
 You can combine this mere tag-detector with text-replacements to point to the actual tags-page.
--- a/examples/Examples.lektorproject
+++ b/examples/Examples.lektorproject
@@ -0,0 +1,5 @@
 [project]
 name = GroupBy Examples
 [packages]
 lektor-groupby = 0.9.3
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -0,0 +1,7 @@
 .PHONY: server clean plugins
 server:
 	lektor server
 clean:
 	lektor clean --yes -v
 plugins:
 	lektor plugins flush-cache && lektor plugins list
--- a/examples/README.md
+++ b/examples/README.md
@@ -0,0 +1,287 @@
 # Usage
 Overview:
 - [quick config example](#quick-config) shows how you can use the plugin config to setup a quick and easy tagging system.
 - [simple example](#simple-example) goes into detail how to use it in your own plugin.
 - [advanced example](#advanced-example) touches on the potentials of the plugin.
 - [Misc](#misc) shows other use-cases.
 ## About
 To use the groupby plugin you have to add an attribute to your model file.
 In our case you can refer to the `models/page.ini` model:
 ```ini
 [fields.tags]
 label = Tags
 type = strings
 testA = true
 testB = true
 [fields.body]
 label = Body
 type = markdown
 testC = true
 ```
 We did define three custom attributes `testA`, `testB`, and `testC`.
 You may add custom attributes to all of the fields.
 It is crucial that the value of the custom attribute is set to true.
 The attribute name is later used for grouping.
 ## Quick config
 Relevant files:
 ```
 configs/groupby.ini
 templates/example-config.html
 ```
 The easiest way to add tags to your site is by defining the `groupby.ini` config file.
 ```ini
 [testA]
 root = /
 slug = config/{key}.html
 template = example-config.html
 split = ' '
 enabled = True
 [testA.fields]
 title = "Tagged: " ~ this.group
 [testA.key_map]
 Blog = News
 ```
 The configuration parameter are:
 1. The section title (`testA`) must be one of the attribute variables we defined in our model.
 2. The `root` parameter (`/`) is the root page of the groupby.
   All results will be placed under this directory, e.g., `/tags/tagname/`.
   If you use `root = /blog`, the results path will be `/blog/tags/tagname/`.
   The groupby plugin will traverse all sub-pages wich contain the attribute `testA`.
 3. The `slug` parameter (`config/{key}.html`) is where the results are placed.
   In our case, the path resolves to `config/tagname.html`.
   The default value is `{attrib}/{key}/index.html` which would resolve to `testA/tagname/index.html`.
   If this field contains `{key}`, it just replaces the value with the group-key.
   In all other cases the field value is evaluated in a jinja context.
 4. The `template`parameter (`example-config.html`) is used to render the results page.
   If no explicit template is set, the default template `groupby-testA.html` will be used.
   Where `testA` is replaced with whatever attribute you chose.
 5. The `split` parameter (`' '`) will be used as string delimiter.
   Fields of type `strings` and `checkboxes` are already lists and don't need splitting.
   The split is only relevant for fields of type `string` or `text`.
   These single-line fields are then expanded to lists as well.
   If you do not provide the `split` option, the whole field value will be used as tagname.
 6. The `enabled` parameter allows you to quickly disable the grouping.
 You can have multiple listeners, e.g., one for `/blog/` and another for `/projects/`.
 Just create as many custom attributes as you like, each having its own section.
 There are two additional config mappings, `.fields` and `.key_map`.
 Key-value pairs in `.fields` will be added as attributes to your grouping.
 You can access them in your template (e.g., `{{this.title}}`).
 All of the `.fields` values are evaluted in a jinja context, so be cautious when using plain strings. 
 The built-in field attributes are:
 - `group`: returned group name, e.g., "A Title?"
 - `key`: slugified group value, e.g., "a-title"
 - `slug`: url path after root node, e.g. "config/a-title.html" (can be `None`)
 - `record`: parent node, e.g., `Page(path="/")`
 - `children`: dictionary of `{record: extras}` pairs
 - `first_child`: first page
 - `first_extra`: first extra
 - `config`: configuration object (see below)
 Without any changes, the `key` value will just be `slugify(group)`.
 However, the other mapping `.key_map` will replace `group` with whatever replacement value is provided in the `.key_map` and then slugified.
 You could, for example, add a `C# = c-sharp` mapping, which would otherwise just be slugified to `c`.
 This is equivalent to `slugify(key_map.get(group))`.
 The `config` attribute contains the values that created the group:
 - `key`: attribute key, e.g., `TestA`
 - `root`: as provided by init, e.g., `/`
 - `slug`: the raw value, e.g., `config/{key}.html`
 - `template`: as provided by init, e.g., `example-config.html`
 - `enabled`: boolean
 - `dependencies`: path to config file (if initialized from config)
 - `fields`: raw values from `TestA.fields`
 - `key_map`: raw values from `TestA.key_map`
 In your template file you have access to the attributes, config, and children (pages):
 ```jinja2
 <h2>{{ this.title }}</h2>
 <p>Key: {{ this.key }}, Attribute: {{ this.config.key }}</p>
 <ul>
 {%- for child in this.children %}
 <li>Page: {{ child.path }}</li>
 {%- endfor %}
 </ul>
 ```
 ## Simple example
 Relevant files:
 ```
 packages/simple-example/lektor_simple.py
 templates/example-simple.html
 ```
 ```python
 def on_groupby_before_build_all(self, groupby, builder, **extra):
    watcher = groupby.add_watcher('testB', {
        'root': '/blog',
        'slug': 'simple/{key}/index.html',
        'template': 'example-simple.html',
    })
    watcher.config.set_key_map({'Foo': 'bar'})
    watcher.config.set_fields({'date': datetime.now()})
    @watcher.grouping(flatten=True)
    def convert_simple_example(args):
        # Yield groups
        value = args.field  # type: list # since model is 'strings' type
        for tag in value:
            yield tag, {'tags_in_page': value}
 ```
 This example is roughly equivalent to the config example above – the parameters of the `groupby.add_watcher` function correspond to the same config parameters.
 Additionally, you can set other types in `set_fields` (all strings are evaluated in jinja context!).
 `@watcher.grouping` sets the callback to generate group keys.
 It has one optional flatten parameter:
 - `flatten` determines how Flow elements are processed.
  If `False`, the callback function is called once per Flow element.
  If `True` (default), the callback is called for all Flow-Blocks of the Flow individually.
  The attribute `testB` can be attached to either the Flow or a Flow-Block regardless.
 The `args` parameter of the `convert_simple_example()` function is a named tuple, it has three attributes:
 1. The `record` points to the `Page` record that contains the tag.
 2. The `key` tuple `(field-key, flow-index, flow-key)` tells which field is processed.
   For Flow types, `flow-index` and `flow-key` are set, otherwise they are `None`.
 3. The `field` value is the content of the processed field.
   The field value is equivalent to the following:
 ```python
 k = args.key
 field = args.record[k.fieldKey].blocks[k.flowIndex].get(k.flowKey)
 ```
 The callback body **can** produce groupings but does not have to.
 If you choose to produce an entry, you have to `yield` a string or tuple pair `(group, extra-info)`.
 `group` is slugified (see above) and then used to combine & cluster pages.
 The `extra-info` (optional) is passed through to your template file.
 You can yield more than one entry per source.
 Or ignore pages if you don't yield anything.
 The template file can access and display the `extra-info`:
 ```jinja2
 <p>Custom field date: {{this.date}}</p>
 <ul>
 {%- for child, extras in this.children.items() -%}
 {%- set etxra = (extras|first).tags_in_page %}
 <li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
 {%- endfor %}
 </ul>
 ```
 ## Advanced example
 Relevant files:
 ```
 configs/advanced.ini
 packages/advanced-example/lektor_advanced.py
 templates/example-advanced.html
 ```
 The following example is similar to the previous one.
 Except that it loads a config file and replaces in-text occurrences of `{{Tagname}}` with `<a href="/tag/">Tagname</a>`.
 ```python
 def on_groupby_before_build_all(self, groupby, builder, **extra):
    # load config
    config = self.get_config()
    regex = config.get('testC.pattern.match')
    try:
        regex = re.compile(regex)
    except Exception as e:
        print('inlinetags.regex not valid: ' + str(e))
        return
    # load config directly (which also tracks dependency)
    watcher = groupby.add_watcher('testC', config)
    @watcher.grouping()
    def convert_replace_example(args):
        # args.field assumed to be Markdown
        obj = args.field.source
        slugify_map = {}  # type Dict[str, str]
        for match in regex.finditer(obj):
            tag = match.group(1)
            key = yield tag
            print('[advanced] slugify:', tag, '->', key)
            slugify_map[tag] = key
        def _fn(match: re.Match) -> str:
            tag = match.group(1)
            return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
        args.field.source = regex.sub(_fn, obj)
 ```
 Notice, `add_watcher` accepts a config file as parameter which keeps also track of dependencies and rebuilds pages when you edit the config file.
 Further, the `yield` call returns the slugified group-key.
 First, you do not need to slugify it yourself and second, potential replacements from `key_map` are already handled.
 For Markdown fields, we can modify the `source` attribute directly.
 All other field types need to be accessed via `args.record` key indirection (see [simple example](#simple-example)).
 ```ini
 [testC]
 root = /
 slug = "advanced/{}/".format(this.key)
 template = example-advanced.html
 [testC.pattern]
 match = {{([^}]{1,32})}}
 ```
 The config file takes the same parameters as the [config example](#quick-config).
 As you can see, `slug` is evaluated in jinja context.
 We introduced a new config option `testC.pattern.match`.
 This regular expression matches `{{` + any string less than 32 characters + `}}`.
 Notice, the parenthesis (`()`) will match only the inner part but the replace function (`re.sub`) will remove the `{{}}`.
 ## Misc
 It was shortly mentioned above that slugs can be `None` (only if manually set to `slug = None`).
 This is useful if you do not want to create subpages but rather an index page containing all groups.
 This can be done in combination with the next use-case:
 ```jinja2
 {%- for x in this|vgroups('TestA', 'TestB', recursive=True)|unique|sort %} 
 <a href="{{ x|url }}">({{ x.group }})</a>
 {%- endfor %} 
 ```
 You can query the groups of any parent node (including those without slug).
 The keys (`'TestA', 'TestB'`) can be omitted which will return all groups of all attributes (you can still filter them with `x.config.key == 'TestC'`).
 Refer to `templates/page.html` for usage.
--- a/examples/configs/advanced.ini
+++ b/examples/configs/advanced.ini
@@ -0,0 +1,15 @@
 [testC]
 root = /
 slug = "advanced/{}/".format(this.key)
 template = example-advanced.html
 [testC.pattern]
 match = {{([^}]{1,32})}}
 [testC.fields]
 desc = "Tag: " ~ this.group ~ ", Key: " ~ this.key
 [testC.key_map]
 Blog = case-sensitive
 Two = three
 three = no-nested-replace
--- a/examples/configs/groupby.ini
+++ b/examples/configs/groupby.ini
@@ -0,0 +1,12 @@
 [testA]
 enabled = True
 root = /
 slug = config/{key}.html
 template = example-config.html
 split = ' '
 [testA.fields]
 title = "Tagged: " ~ this.group
 [testA.key_map]
 Blog = News
--- a/examples/content/blog/contents.lr
+++ b/examples/content/blog/contents.lr
@@ -0,0 +1,9 @@
 _model: blog
 ---
 title: Blog
 ---
 tags:
 Directory
 Blog
 samegroup
--- a/examples/content/blog/first-post/contents.lr
+++ b/examples/content/blog/first-post/contents.lr
@@ -0,0 +1,9 @@
 title: Hello Website
 ---
 body: This is an example blog post. {{Tag}} in {{blog}}
 ---
 tags:
 Initial
 Blog Post
 samegroup
--- a/examples/content/contents.lr
+++ b/examples/content/contents.lr
@@ -0,0 +1,8 @@
 title: GroupBy Examples
 ---
 body: Main body {{tag}} {{Two}}
 ---
 tags:
 Root
 samegroup
--- a/examples/content/projects/contents.lr
+++ b/examples/content/projects/contents.lr
@@ -0,0 +1,9 @@
 title: Projects
 ---
 body:
 This is a list of the projects:
 * Project 1
 * Project 2
 * Project 3
--- a/examples/models/blog-post.ini
+++ b/examples/models/blog-post.ini
@@ -0,0 +1,4 @@
 [model]
 name = Blog Post
 inherits = page
 hidden = yes
--- a/examples/models/blog.ini
+++ b/examples/models/blog.ini
@@ -0,0 +1,7 @@
 [model]
 name = Blog
 inherits = page
 hidden = yes
 [children]
 model = blog-post
--- a/examples/models/page.ini
+++ b/examples/models/page.ini
@@ -0,0 +1,18 @@
 [model]
 name = Page
 label = {{ this.title }}
 [fields.title]
 label = Title
 type = string
 [fields.tags]
 label = Tags
 type = strings
 testA = true
 testB = true
 [fields.body]
 label = Body
 type = markdown
 testC = true
--- a/examples/packages/advanced-example/lektor_advanced.py
+++ b/examples/packages/advanced-example/lektor_advanced.py
@@ -0,0 +1,36 @@
 # -*- coding: utf-8 -*-
 from lektor.pluginsystem import Plugin
 from typing import Generator
 import re
 from lektor_groupby import GroupBy, GroupByCallbackArgs
 class AdvancedGroupByPlugin(Plugin):
    def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
        # load config
        config = self.get_config()
        regex = config.get('testC.pattern.match')
        try:
            regex = re.compile(regex)
        except Exception as e:
            print('inlinetags.regex not valid: ' + str(e))
            return
        # load config directly (which also tracks dependency)
        watcher = groupby.add_watcher('testC', config)
        @watcher.grouping()
        def _replace(args: GroupByCallbackArgs) -> Generator[str, str, None]:
            # args.field assumed to be Markdown
            obj = args.field.source
            slugify_map = {}  # type Dict[str, str]
            for match in regex.finditer(obj):
                tag = match.group(1)
                key = yield tag
                print('[advanced] slugify:', tag, '->', key)
                slugify_map[tag] = key
            def _fn(match: re.Match) -> str:
                tag = match.group(1)
                return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
            args.field.source = regex.sub(_fn, obj)
--- a/examples/packages/advanced-example/setup.py
+++ b/examples/packages/advanced-example/setup.py
@@ -0,0 +1,12 @@
 from setuptools import setup
 setup(
    name='lektor-advanced',
    py_modules=['lektor_advanced'],
    version='1.0',
    entry_points={
        'lektor.plugins': [
            'advanced = lektor_advanced:AdvancedGroupByPlugin',
        ]
    }
 )
--- a/examples/packages/simple-example/lektor_simple.py
+++ b/examples/packages/simple-example/lektor_simple.py
@@ -0,0 +1,33 @@
 # -*- coding: utf-8 -*-
 from lektor.pluginsystem import Plugin
 from typing import Iterator, Tuple
 from datetime import datetime
 from lektor_groupby import GroupBy, GroupByCallbackArgs
 class SimpleGroupByPlugin(Plugin):
    def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
        watcher = groupby.add_watcher('testB', {
            'root': '/blog',
            'slug': 'simple/{key}/index.html',
            'template': 'example-simple.html',
        })
        watcher.config.set_key_map({'Foo': 'bar'})
        watcher.config.set_fields({'date': datetime.now()})
        @watcher.grouping(flatten=True)
        def fn_simple(args: GroupByCallbackArgs) -> Iterator[Tuple[str, dict]]:
            # Yield groups
            value = args.field  # type: list # since model is 'strings' type
            for tag in value:
                yield tag, {'tags_in_page': value}
            # Everything below is just for documentation purposes
            page = args.record  # extract additional info from source
            fieldKey, flowIndex, flowKey = args.key  # or get field index
            if flowIndex is None:
                obj = page[fieldKey]
            else:
                obj = page[fieldKey].blocks[flowIndex].get(flowKey)
            print('[simple] page:', page)
            print('[simple]  obj:', obj)
            print('[simple] ')
--- a/examples/packages/simple-example/setup.py
+++ b/examples/packages/simple-example/setup.py
@@ -0,0 +1,12 @@
 from setuptools import setup
 setup(
    name='lektor-simple',
    py_modules=['lektor_simple'],
    version='1.0',
    entry_points={
        'lektor.plugins': [
            'simple = lektor_simple:SimpleGroupByPlugin',
        ]
    }
 )
--- a/examples/templates/blog-post.html
+++ b/examples/templates/blog-post.html
@@ -0,0 +1,4 @@
 {% extends "page.html" %}
 {% block body %}
  {{ this.body }}
 {% endblock %}
--- a/examples/templates/blog.html
+++ b/examples/templates/blog.html
@@ -0,0 +1,6 @@
 {% extends "page.html" %}
 {% block body %}
  {% for child in this.children %}
    <a href="{{child|url}}">{{ child.title }}</a>
  {% endfor %}
 {% endblock %}
--- a/examples/templates/example-advanced.html
+++ b/examples/templates/example-advanced.html
@@ -0,0 +1,4 @@
 <h2>Path: {{ this | url(absolute=True) }}</h2>
 <p>This is: {{this}}</p>
 <p>Custom field, desc: "{{this.desc}}"</p>
 <p>Children: {{this.children}}</p>
--- a/examples/templates/example-config.html
+++ b/examples/templates/example-config.html
@@ -0,0 +1,9 @@
 <h2>Path: {{ this | url(absolute=True) }}</h2>
 <p>This is: {{this}}</p>
 <p>Group: "{{this.group}}", Key: "{{this.key}}"</p>
 <p>Custom field title: {{this.title}}</p>
 <ul>
 {%- for child in this.children %}
 <li>Child: <a href="{{child|url}}">{{child.title}}</a> ({{child.path}})</li>
 {%- endfor %}
 </ul>
--- a/examples/templates/example-simple.html
+++ b/examples/templates/example-simple.html
@@ -0,0 +1,9 @@
 <h2>Path: {{ this | url(absolute=True) }}</h2>
 <p>This is: {{this}}</p>
 <p>Custom field date: {{this.date}}</p>
 <ul>
 {%- for child, extras in this.children.items() -%}
 {%- set etxra = (extras|first).tags_in_page %}
 <li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
 {%- endfor %}
 </ul>
--- a/examples/templates/page.html
+++ b/examples/templates/page.html
@@ -0,0 +1,29 @@
 <!doctype html>
 <meta charset="utf-8">
 <title>{% block title %}Welcome{% endblock %}</title>
 <style type="text/css">
 header, footer { padding: 1em; background: #DDD; }
 main { margin: 3em; }
 </style>
 <body>
  <header>
    <div>Nav: &nbsp;
      <a href="{{ '/'|url }}">Root</a> &nbsp;
      <a href="{{ '/blog'|url }}">Blog</a> &nbsp;
      <a href="{{ '/projects'|url }}">Projects</a>
    </div>
  </header>
  <main>
    <h2>{{ this.title }}</h2>
    {% block body %}{{ this.body }}{% endblock %}
  </main>
  <footer>
    {%- for k, v in [('testA','Config'),('testB','Simple'),('testC','Advanced')] %}
    <div>{{v}} Tags:
      {%- for x in this|vgroups(k, recursive=True)|unique|sort %} 
        <a href="{{ x|url }}">({{x.key}})</a>
      {%- endfor %} 
    </div>
    {%- endfor %}
  </footer>
 </body>
--- a/lektor_groupby.py
+++ b/lektor_groupby.py
@@ -1,478 +0,0 @@
 # -*- coding: utf-8 -*-
 import lektor.db  # typing
 from lektor.build_programs import BuildProgram
 from lektor.builder import Artifact, Builder  # typing
 from lektor.pluginsystem import Plugin
 from lektor.reporter import reporter
 from lektor.sourceobj import SourceObject, VirtualSourceObject
 from lektor.types.flow import Flow, FlowType
 from lektor.utils import bool_from_string, build_url, prune_file_and_folder
 # for quick config
 from lektor.utils import slugify
 from typing import \
    NewType, NamedTuple, Tuple, Dict, Set, List, Optional, Iterator, Callable
 VPATH = '@groupby'  # potentially unsafe. All matching entries are pruned.
 # -----------------------------------
 #            Typing
 # -----------------------------------
 FieldValue = NewType('FieldValue', object)  # lektor model data-field value
 AttributeKey = NewType('AttributeKey', str)  # attribute of lektor model
 GroupKey = NewType('GroupKey', str)  # key of group-by
 class FieldKeyPath(NamedTuple):
    fieldKey: str
    flowIndex: Optional[int] = None
    flowKey: Optional[str] = None
 class GroupByCallbackArgs(NamedTuple):
    record: lektor.db.Record
    key: FieldKeyPath
    field: FieldValue
 class GroupByCallbackYield(NamedTuple):
    key: GroupKey
    extra: object
 GroupingCallback = Callable[[GroupByCallbackArgs],
                            Iterator[GroupByCallbackYield]]
 class GroupProducer(NamedTuple):
    attribute: AttributeKey
    func: GroupingCallback
    flatten: bool = True
    slug: Optional[str] = None
    template: Optional[str] = None
    dependency: Optional[str] = None
 class GroupComponent(NamedTuple):
    record: lektor.db.Record
    extra: object
 # -----------------------------------
 #            Actual logic
 # -----------------------------------
 class GroupBySource(VirtualSourceObject):
    '''
    Holds information for a single group/cluster.
    This object is accessible in your template file.
    Attributes: record, attribute, group, children, template, slug
    :DEFAULTS:
    template: "groupby-attribute.html"
    slug: "{attrib}/{group}/index.html"
    '''
    def __init__(
        self,
        record: lektor.db.Record,
        attribute: AttributeKey,
        group: GroupKey,
        children: List[GroupComponent] = [],
        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
        template: Optional[str] = None,  # default: "groupby-attribute.html"
        dependency: Optional[str] = None
    ):
        super().__init__(record)
        self.attribute = attribute
        self.group = group
        self.children = children
        self.template = template or 'groupby-{}.html'.format(self.attribute)
        self.dependency = dependency
        # custom user path
        slug = slug or '{attrib}/{group}/index.html'
        slug = slug.replace('{attrib}', self.attribute)
        slug = slug.replace('{group}', self.group)
        if slug.endswith('/index.html'):
            slug = slug[:-10]
        self.slug = slug
    @property
    def path(self) -> str:
        # Used in VirtualSourceInfo, used to prune VirtualObjects
        return build_url([self.record.path, VPATH, self.attribute, self.group])
    @property
    def url_path(self) -> str:
        return build_url([self.record.path, self.slug])
    def iter_source_filenames(self) -> Iterator[str]:
        if self.dependency:
            yield self.dependency
        for record, _ in self.children:
            yield from record.iter_source_filenames()
    def __str__(self) -> str:
        txt = '<GroupBySource'
        for x in ['attribute', 'group', 'template', 'slug']:
            txt += ' {}="{}"'.format(x, getattr(self, x))
        return txt + ' children={}>'.format(len(self.children))
 class GroupByBuildProgram(BuildProgram):
    ''' Generates Build-Artifacts and write files. '''
    def produce_artifacts(self) -> None:
        url = self.source.url_path
        if url.endswith('/'):
            url += 'index.html'
        self.declare_artifact(url, sources=list(
            self.source.iter_source_filenames()))
        GroupByPruner.track(url)
    def build_artifact(self, artifact: Artifact) -> None:
        self.source.pad.db.track_record_dependency(self.source)
        artifact.render_template_into(self.source.template, this=self.source)
 # -----------------------------------
 #              Helper
 # -----------------------------------
 class GroupByPruner:
    '''
    Static collector for build-artifact urls.
    All non-tracked VPATH-urls will be pruned after build.
    '''
    _cache: Set[str] = set()
    # Note: this var is static or otherwise two instances of
    #       GroupByCreator would prune each others artifacts.
    @classmethod
    def track(cls, url: str) -> None:
        cls._cache.add(url.lstrip('/'))
    @classmethod
    def prune(cls, builder: Builder) -> None:
        ''' Remove previously generated, unreferenced Artifacts. '''
        dest_path = builder.destination_path
        con = builder.connect_to_database()
        try:
            with builder.new_build_state() as build_state:
                for url, file in build_state.iter_artifacts():
                    if url.lstrip('/') in cls._cache:
                        continue  # generated in this build-run
                    infos = build_state.get_artifact_dependency_infos(url, [])
                    for v_path, _ in infos:
                        if VPATH not in v_path:
                            continue  # we only care about groupby Virtuals
                        reporter.report_pruned_artifact(url)
                        prune_file_and_folder(file.filename, dest_path)
                        build_state.remove_artifact(url)
                        break  # there is only one VPATH-entry per source
        finally:
            con.close()
        cls._cache.clear()
 # -----------------------------------
 #           Main Component
 # -----------------------------------
 class GroupByCreator:
    '''
    Process all children with matching conditions under specified page.
    Creates a grouping of pages with similar (self-defined) attributes.
    The grouping is performed only once per build (or manually invoked).
    '''
    def __init__(self):
        self._flows: Dict[AttributeKey, Dict[str, Set[str]]] = {}
        self._models: Dict[AttributeKey, Dict[str, Dict[str, str]]] = {}
        self._func: Dict[str, Set[GroupProducer]] = {}
        self._resolve_map: Dict[str, GroupBySource] = {}  # only for server
        self._watched_once: Set[GroupingCallback] = set()
    # --------------
    #   Initialize
    # --------------
    def initialize(self, db: lektor.db):
        self._flows.clear()
        self._models.clear()
        self._resolve_map.clear()
        for prod_list in self._func.values():
            for producer in prod_list:
                self._register(db, producer.attribute)
    def _register(self, db: lektor.db, attrib: AttributeKey) -> None:
        ''' Preparation: find models and flow-models which contain attrib '''
        if attrib in self._flows or attrib in self._models:
            return  # already added
        # find flow blocks with attrib
        _flows = {}  # Dict[str, Set[str]]
        for key, flow in db.flowblocks.items():
            tmp1 = set(f.name for f in flow.fields
                       if bool_from_string(f.options.get(attrib, False)))
            if tmp1:
                _flows[key] = tmp1
        # find models with attrib or flow-blocks containing attrib
        _models = {}  # Dict[str, Dict[str, str]]
        for key, model in db.datamodels.items():
            tmp2 = {}  # Dict[str, str]
            for field in model.fields:
                if bool_from_string(field.options.get(attrib, False)):
                    tmp2[field.name] = '*'  # include all children
                elif isinstance(field.type, FlowType):
                    if any(x in _flows for x in field.type.flow_blocks):
                        tmp2[field.name] = '?'  # only some flow blocks
            if tmp2:
                _models[key] = tmp2
        self._flows[attrib] = _flows
        self._models[attrib] = _models
    # ----------------
    #   Add Observer
    # ----------------
    def watch(
        self,
        root: str,
        attrib: AttributeKey, *,
        flatten: bool = True,  # if False, dont explode FlowType
        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
        template: Optional[str] = None,  # default: "groupby-attrib.html"
        dependency: Optional[str] = None
    ) -> Callable[[GroupingCallback], None]:
        '''
        Decorator to subscribe to attrib-elements. Converter for groupby().
        Refer to groupby() for further details.
        (record, field-key, field) -> (group-key, extra-info)
        :DEFAULTS:
        template: "groupby-attrib.html"
        slug: "{attrib}/{group}/index.html"
        '''
        root = root.rstrip('/') + '/'
        def _decorator(fn: GroupingCallback):
            if root not in self._func:
                self._func[root] = set()
            self._func[root].add(
                GroupProducer(attrib, fn, flatten, template, slug, dependency))
        return _decorator
    def watch_once(self, *args, **kwarg) -> Callable[[GroupingCallback], None]:
        ''' Same as watch() but listener is auto removed after build. '''
        def _decorator(fn: GroupingCallback):
            self._watched_once.add(fn)
            self.watch(*args, **kwarg)(fn)
        return _decorator
    def remove_watch_once(self) -> None:
        ''' Remove all watch-once listeners. '''
        for k, v in self._func.items():
            not_once = {x for x in v if x.func not in self._watched_once}
            self._func[k] = not_once
        self._watched_once.clear()
    # ----------
    #   Helper
    # ----------
    def iter_record_fields(
        self,
        source: lektor.db.Record,
        attrib: AttributeKey,
        flatten: bool = False
    ) -> Iterator[Tuple[FieldKeyPath, FieldValue]]:
        ''' Enumerate all fields of a lektor.db.Record with attrib = True '''
        assert isinstance(source, lektor.db.Record)
        _flows = self._flows.get(attrib, {})
        _models = self._models.get(attrib, {})
        for r_key, subs in _models.get(source.datamodel.id, {}).items():
            if subs == '*':  # either normal field or flow type (all blocks)
                field = source[r_key]
                if flatten and isinstance(field, Flow):
                    for i, flow in enumerate(field.blocks):
                        flowtype = flow['_flowblock']
                        for f_key, block in flow._data.items():
                            if f_key.startswith('_'):  # e.g., _flowblock
                                continue
                            yield FieldKeyPath(r_key, i, f_key), block
                else:
                    yield FieldKeyPath(r_key), field
            else:  # always flow type (only some blocks)
                for i, flow in enumerate(source[r_key].blocks):
                    flowtype = flow['_flowblock']
                    for f_key in _flows.get(flowtype, []):
                        yield FieldKeyPath(r_key, i, f_key), flow[f_key]
    def groupby(
        self,
        attrib: AttributeKey,
        root: lektor.db.Record,
        func: GroupingCallback,
        flatten: bool = False,
        incl_attachments: bool = True
    ) -> Dict[GroupKey, List[GroupComponent]]:
        '''
        Traverse selected root record with all children and group by func.
        Func is called with (record, FieldKeyPath, FieldValue).
        Func may yield one or more (group-key, extra-info) tuples.
        return {'group-key': [(record, extra-info), ...]}
        '''
        assert callable(func), 'no GroupingCallback provided'
        assert isinstance(root, lektor.db.Record)
        tmap = {}  # type: Dict[GroupKey, List[GroupComponent]]
        recursive_list = [root]  # type: List[lektor.db.Record]
        while recursive_list:
            record = recursive_list.pop()
            if hasattr(record, 'children'):
                # recursive_list += record.children
                recursive_list.extend(record.children)
            if incl_attachments and hasattr(record, 'attachments'):
                # recursive_list += record.attachments
                recursive_list.extend(record.attachments)
            for key, field in self.iter_record_fields(record, attrib, flatten):
                for ret in func(GroupByCallbackArgs(record, key, field)) or []:
                    assert isinstance(ret, (tuple, list)), \
                        'Must return tuple (group-key, extra-info)'
                    group_key, extras = ret
                    if group_key not in tmap:
                        tmap[group_key] = []
                    tmap[group_key].append(GroupComponent(record, extras))
        return tmap
    # -----------------
    #   Create groups
    # -----------------
    def should_process(self, node: SourceObject) -> bool:
        ''' Check if record path is being watched. '''
        return isinstance(node, lektor.db.Record) \
            and node.url_path in self._func
    def make_cluster(self, root: lektor.db.Record) -> Iterator[GroupBySource]:
        ''' Group by attrib and build Artifacts. '''
        assert isinstance(root, lektor.db.Record)
        for attr, fn, fl, temp, slug, dep in self._func.get(root.url_path, []):
            groups = self.groupby(attr, root, func=fn, flatten=fl)
            for group_key, children in groups.items():
                obj = GroupBySource(root, attr, group_key, children,
                                    template=temp, slug=slug, dependency=dep)
                self.track_dev_server_path(obj)
                yield obj
    # ------------------
    #   Path resolving
    # ------------------
    def resolve_virtual_path(
        self, node: SourceObject, pieces: List[str]
    ) -> Optional[GroupBySource]:
        ''' Given a @VPATH/attrib/groupkey path, determine url path. '''
        if len(pieces) >= 2:
            attrib: AttributeKey = pieces[0]  # type: ignore[assignment]
            group: GroupKey = pieces[1]  # type: ignore[assignment]
            for attr, _, _, _, slug, _ in self._func.get(node.url_path, []):
                if attr == attrib:
                    # TODO: do we need to provide the template too?
                    return GroupBySource(node, attr, group, slug=slug)
        return None
    def track_dev_server_path(self, sender: GroupBySource) -> None:
        ''' Dev server only: Add target path to reverse artifact url lookup '''
        self._resolve_map[sender.url_path] = sender
    def resolve_dev_server_path(
        self, node: SourceObject, pieces: List[str]
    ) -> Optional[GroupBySource]:
        ''' Dev server only: Resolve actual url to virtual obj. '''
        return self._resolve_map.get(build_url([node.url_path] + pieces))
 # -----------------------------------
 #           Plugin Entry
 # -----------------------------------
 class GroupByPlugin(Plugin):
    name = 'GroupBy Plugin'
    description = 'Cluster arbitrary records with field attribute keyword.'
    def on_setup_env(self, **extra):
        self.creator = GroupByCreator()
        self.env.add_build_program(GroupBySource, GroupByBuildProgram)
        # let other plugins register their @groupby.watch functions
        self.emit('init', groupby=self.creator, **extra)
        # resolve /tag/rss/ -> /tag/rss/index.html (local server only)
        @self.env.urlresolver
        def groupby_path_resolver(node, pieces):
            if self.creator.should_process(node):
                return self.creator.resolve_dev_server_path(node, pieces)
        # use VPATH in templates: {{ '/@groupby/attrib/group' | url }}
        @self.env.virtualpathresolver(VPATH.lstrip('@'))
        def groupby_virtualpath_resolver(node, pieces):
            if self.creator.should_process(node):
                return self.creator.resolve_virtual_path(node, pieces)
        # injection to generate GroupBy nodes when processing artifacts
        # @self.env.generator
        # def groupby_generator(node):
        #     if self.creator.should_process(node):
        #         yield from self.creator.make_cluster(node)
    def _quick_config(self):
        config = self.get_config()
        for attrib in config.sections():
            sect = config.section_as_dict(attrib)
            root = sect.get('root', '/')
            slug = sect.get('slug')
            temp = sect.get('template')
            split = sect.get('split')
            @self.creator.watch_once(root, attrib, template=temp, slug=slug,
                                     dependency=self.config_filename)
            def _fn(args):
                val = args.field
                if isinstance(val, str):
                    val = val.split(split) if split else [val]  # make list
                if isinstance(val, list):
                    for tag in val:
                        yield slugify(tag), tag
    def on_before_build_all(self, builder, **extra):
        # let other plugins register their @groupby.watch_once functions
        self.emit('init-once', groupby=self.creator, builder=builder, **extra)
        # load config file quick listeners (before initialize!)
        self._quick_config()
        # parse all models to detect attribs of listeners
        self.creator.initialize(builder.pad.db)
    def on_before_build(self, builder, build_state, source, prog, **extra):
        # Injection to create GroupBy nodes before parent node is built.
        # Use this callback (not @generator) to modify parent beforehand.
        # Relevant for the root page which is otherwise build before GroupBy.
        if self.creator.should_process(source):
            for vobj in self.creator.make_cluster(source):
                builder.build(vobj)
    def on_after_build_all(self, builder, **extra):
        # remove all quick listeners (will be added again in the next build)
        self.creator.remove_watch_once()
    def on_after_prune(self, builder, **extra):
        # TODO: find better way to prune unreferenced elements
        GroupByPruner.prune(builder)
--- a/lektor_groupby/init.py
+++ b/lektor_groupby/init.py
@@ -0,0 +1,4 @@
 from .config import Config  # noqa: F401
 from .groupby import GroupBy  # noqa: F401
 from .plugin import GroupByPlugin  # noqa: F401
 from .watcher import GroupByCallbackArgs  # noqa: F401
--- a/lektor_groupby/config.py
+++ b/lektor_groupby/config.py
@@ -0,0 +1,87 @@
 from inifile import IniFile
 from lektor.utils import slugify
 from typing import Set, Dict, Optional, Union, Any
 AnyConfig = Union['Config', IniFile, Dict]
 class Config:
    '''
    Holds information for GroupByWatcher and GroupBySource.
    This object is accessible in your template file ({{this.config}}).
    Available attributes:
    key, root, slug, template, enabled, dependencies, fields, key_map
    '''
    def __init__(
        self,
        key: str, *,
        root: Optional[str] = None,  # default: "/"
        slug: Optional[str] = None,  # default: "{attr}/{group}/index.html"
        template: Optional[str] = None,  # default: "groupby-{attr}.html"
    ) -> None:
        self.key = key
        self.root = (root or '/').rstrip('/') or '/'
        self.slug = slug or (key + '/{key}/')  # key = GroupBySource.key
        self.template = template or f'groupby-{self.key}.html'
        # editable after init
        self.enabled = True
        self.dependencies = set()  # type: Set[str]
        self.fields = {}  # type: Dict[str, Any]
        self.key_map = {}  # type: Dict[str, str]
    def slugify(self, k: str) -> str:
        ''' key_map replace and slugify. '''
        return slugify(self.key_map.get(k, k))  # type: ignore[no-any-return]
    def set_fields(self, fields: Optional[Dict[str, Any]]) -> None:
        '''
        The fields dict is a mapping of attrib = Expression values.
        Each dict key will be added to the GroupBySource virtual object.
        Each dict value is passed through jinja context first.
        '''
        self.fields = fields or {}
    def set_key_map(self, key_map: Optional[Dict[str, str]]) -> None:
        ''' This mapping replaces group keys before slugify. '''
        self.key_map = key_map or {}
    def __repr__(self) -> str:
        txt = '<GroupByConfig'
        for x in ['key', 'root', 'slug', 'template', 'enabled']:
            txt += ' {}="{}"'.format(x, getattr(self, x))
        txt += f' fields="{", ".join(self.fields)}"'
        return txt + '>'
    @staticmethod
    def from_dict(key: str, cfg: Dict[str, str]) -> 'Config':
        ''' Set config fields manually. Allowed: key, root, slug, template. '''
        return Config(
            key=key,
            root=cfg.get('root'),
            slug=cfg.get('slug'),
            template=cfg.get('template'),
        )
    @staticmethod
    def from_ini(key: str, ini: IniFile) -> 'Config':
        ''' Read and parse ini file. Also adds dependency tracking. '''
        cfg = ini.section_as_dict(key)  # type: Dict[str, str]
        conf = Config.from_dict(key, cfg)
        conf.enabled = ini.get_bool(key + '.enabled', True)
        conf.dependencies.add(ini.filename)
        conf.set_fields(ini.section_as_dict(key + '.fields'))
        conf.set_key_map(ini.section_as_dict(key + '.key_map'))
        return conf
    @staticmethod
    def from_any(key: str, config: AnyConfig) -> 'Config':
        assert isinstance(config, (Config, IniFile, Dict))
        if isinstance(config, Config):
            return config
        elif isinstance(config, IniFile):
            return Config.from_ini(key, config)
        elif isinstance(config, Dict):
            return Config.from_dict(key, config)
--- a/lektor_groupby/groupby.py
+++ b/lektor_groupby/groupby.py
@@ -0,0 +1,116 @@
 from lektor.builder import Builder, PathCache
 from lektor.db import Record
 from lektor.sourceobj import SourceObject
 from lektor.utils import build_url
 from typing import Set, Dict, List, Optional, Tuple
 from .vobj import GroupBySource
 from .config import Config, AnyConfig
 from .watcher import Watcher
 class GroupBy:
    '''
    Process all children with matching conditions under specified page.
    Creates a grouping of pages with similar (self-defined) attributes.
    The grouping is performed only once per build.
    '''
    def __init__(self) -> None:
        self._watcher = []  # type: List[Watcher]
        self._results = []  # type: List[GroupBySource]
        self._resolver = {}  # type: Dict[str, Tuple[str, Config]]
    # ----------------
    #   Add observer
    # ----------------
    def add_watcher(self, key: str, config: AnyConfig) -> Watcher:
        ''' Init Config and add to watch list. '''
        w = Watcher(Config.from_any(key, config))
        self._watcher.append(w)
        return w
    # -----------
    #   Builder
    # -----------
    def clear_previous_results(self) -> None:
        ''' Reset prvious results. Must be called before each build. '''
        self._watcher.clear()
        self._results.clear()
        self._resolver.clear()
    def get_dependencies(self) -> Set[str]:
        deps = set()  # type: Set[str]
        for w in self._watcher:
            deps.update(w.config.dependencies)
        return deps
    def make_cluster(self, builder: Builder) -> None:
        ''' Iterate over all children and perform groupby. '''
        # remove disabled watchers
        self._watcher = [w for w in self._watcher if w.config.enabled]
        if not self._watcher:
            return
        # initialize remaining (enabled) watchers
        for w in self._watcher:
            w.initialize(builder.pad.db)
        # iterate over whole build tree
        queue = builder.pad.get_all_roots()  # type: List[SourceObject]
        while queue:
            record = queue.pop()
            self.queue_now(record)
            if hasattr(record, 'attachments'):
                queue.extend(record.attachments)  # type: ignore[attr-defined]
            if hasattr(record, 'children'):
                queue.extend(record.children)  # type: ignore[attr-defined]
        # build artifacts
        for w in self._watcher:
            root = builder.pad.get(w.config.root)
            for vobj in w.iter_sources(root):
                self._results.append(vobj)
                if vobj.slug:
                    self._resolver[vobj.url_path] = (vobj.group, w.config)
        self._watcher.clear()
    def queue_now(self, node: SourceObject) -> None:
        ''' Process record immediatelly (No-Op if already processed). '''
        if isinstance(node, Record):
            for w in self._watcher:
                if w.should_process(node):
                    w.process(node)
    def build_all(self, builder: Builder) -> None:
        ''' Create virtual objects and build sources. '''
        path_cache = PathCache(builder.env)
        for vobj in self._results:
            if vobj.slug:
                builder.build(vobj, path_cache)
        del path_cache
        self._results.clear()  # garbage collect weak refs
    # -----------------
    #   Path resolver
    # -----------------
    def resolve_dev_server_path(self, node: SourceObject, pieces: List[str]) \
            -> Optional[GroupBySource]:
        ''' Dev server only: Resolves path/ -> path/index.html '''
        if isinstance(node, Record):
            rv = self._resolver.get(build_url([node.url_path] + pieces))
            if rv:
                return GroupBySource(node, group=rv[0], config=rv[1])
        return None
    def resolve_virtual_path(self, node: SourceObject, pieces: List[str]) \
            -> Optional[GroupBySource]:
        ''' Admin UI only: Prevent server error and null-redirect. '''
        if isinstance(node, Record) and len(pieces) >= 2:
            path = node['_path']  # type: str
            key, grp, *_ = pieces
            for group, conf in self._resolver.values():
                if key == conf.key and path == conf.root:
                    if conf.slugify(group) == grp:
                        return GroupBySource(node, group, conf)
        return None
--- a/lektor_groupby/plugin.py
+++ b/lektor_groupby/plugin.py
@@ -0,0 +1,67 @@
 from lektor.builder import Builder  # typing
 from lektor.pluginsystem import Plugin  # subclass
 from lektor.sourceobj import SourceObject  # typing
 from typing import List, Optional, Iterator
 from .vobj import GroupBySource, GroupByBuildProgram, VPATH
 from .groupby import GroupBy
 from .pruner import prune
 from .watcher import GroupByCallbackArgs  # typing
 class GroupByPlugin(Plugin):
    name = 'GroupBy Plugin'
    description = 'Cluster arbitrary records with field attribute keyword.'
    def on_setup_env(self, **extra: object) -> None:
        self.creator = GroupBy()
        self.env.add_build_program(GroupBySource, GroupByBuildProgram)
        self.env.jinja_env.filters.update(vgroups=GroupBySource.of_record)
        # resolve /tag/rss/ -> /tag/rss/index.html (local server only)
        @self.env.urlresolver
        def a(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
            return self.creator.resolve_dev_server_path(node, parts)
        # resolve virtual objects in admin UI
        @self.env.virtualpathresolver(VPATH.lstrip('@'))
        def b(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
            return self.creator.resolve_virtual_path(node, parts)
    def _load_quick_config(self) -> None:
        ''' Load config file quick listeners. '''
        config = self.get_config()
        for key in config.sections():
            if '.' in key:  # e.g., key.fields and key.key_map
                continue
            watcher = self.creator.add_watcher(key, config)
            split = config.get(key + '.split')  # type: str
            @watcher.grouping()
            def _fn(args: GroupByCallbackArgs) -> Iterator[str]:
                val = args.field
                if isinstance(val, str):
                    val = val.split(split) if split else [val]  # make list
                if isinstance(val, list):
                    yield from val
    def on_before_build_all(self, builder: Builder, **extra: object) -> None:
        self.creator.clear_previous_results()
        self._load_quick_config()
        # let other plugins register their @groupby.watch functions
        self.emit('before-build-all', groupby=self.creator, builder=builder)
        self.config_dependencies = self.creator.get_dependencies()
        self.creator.make_cluster(builder)
    def on_before_build(self, source: SourceObject, **extra: object) -> None:
        # before-build may be called before before-build-all (issue #1017)
        # make sure it is evaluated immediatelly
        self.creator.queue_now(source)
    def on_after_build_all(self, builder: Builder, **extra: object) -> None:
        self.creator.build_all(builder)
    def on_after_prune(self, builder: Builder, **extra: object) -> None:
        # TODO: find a better way to prune unreferenced elements
        prune(builder, VPATH)
--- a/lektor_groupby/pruner.py
+++ b/lektor_groupby/pruner.py
@@ -0,0 +1,39 @@
 '''
 Static collector for build-artifact urls.
 All non-tracked VPATH-urls will be pruned after build.
 '''
 from lektor.builder import Builder  # typing
 from lektor.reporter import reporter  # report_pruned_artifact
 from lektor.utils import prune_file_and_folder
 _cache = set()
 # Note: this var is static or otherwise two instances of
 #       this module would prune each others artifacts.
 def track_not_prune(url: str) -> None:
    ''' Add url to build cache to prevent pruning. '''
    _cache.add(url.lstrip('/'))
 def prune(builder: Builder, vpath: str) -> None:
    ''' Remove previously generated, unreferenced Artifacts. '''
    vpath = '@' + vpath.lstrip('@')  # just in case of user error
    dest_path = builder.destination_path
    con = builder.connect_to_database()
    try:
        with builder.new_build_state() as build_state:
            for url, file in build_state.iter_artifacts():
                if url.lstrip('/') in _cache:
                    continue  # generated in this build-run
                infos = build_state.get_artifact_dependency_infos(url, [])
                for artifact_name, _ in infos:
                    if vpath not in artifact_name:
                        continue  # we only care about our Virtuals
                    reporter.report_pruned_artifact(url)
                    prune_file_and_folder(file.filename, dest_path)
                    build_state.remove_artifact(url)
                    break  # there is only one VPATH-entry per source
    finally:
        con.close()
    _cache.clear()
--- a/lektor_groupby/util.py
+++ b/lektor_groupby/util.py
@@ -0,0 +1,11 @@
 from lektor.reporter import reporter, style
 def report_config_error(key: str, field: str, val: str, e: Exception) -> None:
    ''' Send error message to Lektor reporter. Indicate which field is bad. '''
    msg = '[ERROR] invalid config for [{}.{}] = "{}",  Error: {}'.format(
        key, field, val, repr(e))
    try:
        reporter._write_line(style(msg, fg='red'))
    except Exception:
        print(msg)  # fallback in case Lektor API changes
--- a/lektor_groupby/vobj.py
+++ b/lektor_groupby/vobj.py
@@ -0,0 +1,197 @@
 from lektor.build_programs import BuildProgram  # subclass
 from lektor.builder import Artifact  # typing
 from lektor.context import get_ctx
 from lektor.db import Record  # typing
 from lektor.environment import Expression
 from lektor.sourceobj import VirtualSourceObject  # subclass
 from lektor.utils import build_url
 from typing import Dict, List, Any, Optional, Iterator
 from weakref import WeakSet
 from .config import Config
 from .pruner import track_not_prune
 from .util import report_config_error
 VPATH = '@groupby'  # potentially unsafe. All matching entries are pruned.
 # -----------------------------------
 #           VirtualSource
 # -----------------------------------
 class GroupBySource(VirtualSourceObject):
    '''
    Holds information for a single group/cluster.
    This object is accessible in your template file.
    Attributes: record, key, group, slug, children, config
    '''
    def __init__(
        self,
        record: Record,
        group: str,
        config: Config,
        children: Optional[Dict[Record, List[Any]]] = None,
    ) -> None:
        super().__init__(record)
        self.key = config.slugify(group)
        self.group = group
        self.config = config
        # evaluate slug Expression
        if config.slug and '{key}' in config.slug:
            self.slug = config.slug.replace('{key}', self.key)
        else:
            self.slug = self._eval(config.slug, field='slug')
            assert self.slug != Ellipsis, 'invalid config: ' + config.slug
        if self.slug and self.slug.endswith('/index.html'):
            self.slug = self.slug[:-10]
        # make sure children are on the same pad
        self._children = {}  # type: Dict[Record, List[Any]]
        for child, extras in (children or {}).items():
            if child.pad != record.pad:
                child = record.pad.get(child.path)
            self._children[child] = extras
        self._reverse_reference_records()
        # extra fields
        for attr, expr in config.fields.items():
            setattr(self, attr, self._eval(expr, field='fields.' + attr))
    def _eval(self, value: Any, *, field: str) -> Any:
        ''' Internal only: evaluates Lektor config file field expression. '''
        if not isinstance(value, str):
            return value
        pad = self.record.pad
        alt = self.record.alt
        try:
            return Expression(pad.env, value).evaluate(pad, this=self, alt=alt)
        except Exception as e:
            report_config_error(self.config.key, field, value, e)
            return Ellipsis
    # ---------------------
    #   Lektor properties
    # ---------------------
    @property
    def path(self) -> str:
        # Used in VirtualSourceInfo, used to prune VirtualObjects
        return f'{self.record.path}{VPATH}/{self.config.key}/{self.key}'
    @property
    def url_path(self) -> str:
        # Actual path to resource as seen by the browser
        return build_url([self.record.path, self.slug])  # slug can be None!
    def iter_source_filenames(self) -> Iterator[str]:
        ''' Enumerate all dependencies '''
        if self.config.dependencies:
            yield from self.config.dependencies
        for record in self._children:
            yield from record.iter_source_filenames()
    # -----------------------
    #   Properties & Helper
    # -----------------------
    @property
    def children(self) -> Dict[Record, List[Any]]:
        ''' Returns dict with page record key and (optional) extra value. '''
        return self._children
    @property
    def first_child(self) -> Optional[Record]:
        ''' Returns first referencing page record. '''
        if self._children:
            return iter(self._children).__next__()
        return None
    @property
    def first_extra(self) -> Optional[Any]:
        ''' Returns first additional / extra info object of first page. '''
        if not self._children:
            return None
        val = iter(self._children.values()).__next__()
        return val[0] if val else None
    def __getitem__(self, key: str) -> Any:
        # Used for virtual path resolver and |sort(attribute="x") filter
        if key in ('_path', '_alt'):
            return getattr(self, key[1:])
        if hasattr(self, key):
            return getattr(self, key)
        return None
    def __lt__(self, other: 'GroupBySource') -> bool:
        # Used for |sort filter ("group" is the provided original string)
        return self.group < other.group
    def __eq__(self, other: object) -> bool:
        # Used for |unique filter
        if self is other:
            return True
        return isinstance(other, GroupBySource) and \
            self.path == other.path and self.slug == other.slug
    def __hash__(self) -> int:
        # Used for hashing in set and dict
        return hash((self.path, self.slug))
    def __repr__(self) -> str:
        return '<GroupBySource path="{}" children={}>'.format(
            self.path, len(self._children))
    # ---------------------
    #   Reverse Reference
    # ---------------------
    def _reverse_reference_records(self) -> None:
        ''' Attach self to page records. '''
        for child in self._children:
            if not hasattr(child, '_vgroups'):
                child._vgroups = WeakSet()  # type: ignore[attr-defined]
            child._vgroups.add(self)  # type: ignore[attr-defined]
    @staticmethod
    def of_record(
        record: Record,
        *keys: str,
        recursive: bool = False
    ) -> Iterator['GroupBySource']:
        ''' Extract all referencing groupby virtual objects from a page. '''
        ctx = get_ctx()
        # manage dependencies
        if ctx:
            for dep in ctx.env.plugins['groupby'].config_dependencies:
                ctx.record_dependency(dep)
        # find groups
        proc_list = [record]
        while proc_list:
            page = proc_list.pop(0)
            if recursive and hasattr(page, 'children'):
                proc_list.extend(page.children)  # type: ignore[attr-defined]
            if not hasattr(page, '_vgroups'):
                continue
            for vobj in page._vgroups:  # type: ignore[attr-defined]
                if not keys or vobj.config.key in keys:
                    yield vobj
 # -----------------------------------
 #           BuildProgram
 # -----------------------------------
 class GroupByBuildProgram(BuildProgram):
    ''' Generate Build-Artifacts and write files. '''
    def produce_artifacts(self) -> None:
        url = self.source.url_path
        if url.endswith('/'):
            url += 'index.html'
        self.declare_artifact(url, sources=list(
            self.source.iter_source_filenames()))
        track_not_prune(url)
    def build_artifact(self, artifact: Artifact) -> None:
        get_ctx().record_virtual_dependency(self.source)
        artifact.render_template_into(
            self.source.config.template, this=self.source)
--- a/lektor_groupby/watcher.py
+++ b/lektor_groupby/watcher.py
@@ -0,0 +1,199 @@
 from lektor.db import Database, Record  # typing
 from lektor.types.flow import Flow, FlowType
 from lektor.utils import bool_from_string
 from typing import Set, Dict, List, Tuple, Any, Union, NamedTuple
 from typing import Optional, Callable, Iterable, Iterator, Generator
 from .vobj import GroupBySource
 from .config import Config
 # -----------------------------------
 #              Typing
 # -----------------------------------
 class FieldKeyPath(NamedTuple):
    fieldKey: str
    flowIndex: Optional[int] = None
    flowKey: Optional[str] = None
 class GroupByCallbackArgs(NamedTuple):
    record: Record
    key: FieldKeyPath
    field: Any  # lektor model data-field value
 GroupingCallback = Callable[[GroupByCallbackArgs], Union[
    Iterator[Union[str, Tuple[str, Any]]],
    Generator[Union[str, Tuple[str, Any]], Optional[str], None],
 ]]
 # -----------------------------------
 #            ModelReader
 # -----------------------------------
 class GroupByModelReader:
    ''' Find models and flow-models which contain attribute '''
    def __init__(self, db: Database, attrib: str) -> None:
        self._flows = {}  # type: Dict[str, Set[str]]
        self._models = {}  # type: Dict[str, Dict[str, str]]
        # find flow blocks containing attribute
        for key, flow in db.flowblocks.items():
            tmp1 = set(f.name for f in flow.fields
                       if bool_from_string(f.options.get(attrib, False)))
            if tmp1:
                self._flows[key] = tmp1
        # find models and flow-blocks containing attribute
        for key, model in db.datamodels.items():
            tmp2 = {}  # Dict[str, str]
            for field in model.fields:
                if bool_from_string(field.options.get(attrib, False)):
                    tmp2[field.name] = '*'  # include all children
                elif isinstance(field.type, FlowType) and self._flows:
                    # only processed if at least one flow has attrib
                    fbs = field.type.flow_blocks
                    # if fbs == None, all flow-blocks are allowed
                    if fbs is None or any(x in self._flows for x in fbs):
                        tmp2[field.name] = '?'  # only some flow blocks
            if tmp2:
                self._models[key] = tmp2
    def read(
        self,
        record: Record,
        flatten: bool = False
    ) -> Iterator[Tuple[FieldKeyPath, Any]]:
        '''
        Enumerate all fields of a Record with attrib = True.
        Flows are either returned directly (flatten=False) or
        expanded so that each flow-block is yielded (flatten=True)
        '''
        assert isinstance(record, Record)
        for r_key, subs in self._models.get(record.datamodel.id, {}).items():
            if subs == '*':  # either normal field or flow type (all blocks)
                field = record[r_key]
                if flatten and isinstance(field, Flow):
                    for i, flow in enumerate(field.blocks):
                        flowtype = flow['_flowblock']
                        for f_key, block in flow._data.items():
                            if f_key.startswith('_'):  # e.g., _flowblock
                                continue
                            yield FieldKeyPath(r_key, i, f_key), block
                else:
                    yield FieldKeyPath(r_key), field
            else:  # always flow type (only some blocks)
                for i, flow in enumerate(record[r_key].blocks):
                    flowtype = flow['_flowblock']
                    for f_key in self._flows.get(flowtype, []):
                        yield FieldKeyPath(r_key, i, f_key), flow[f_key]
 # -----------------------------------
 #               State
 # -----------------------------------
 class GroupByState:
    ''' Store and update a groupby build state. {group: {record: [extras]}} '''
    def __init__(self) -> None:
        self.state = {}  # type: Dict[str, Dict[Record, List[Any]]]
        self._processed = set()  # type: Set[Record]
    def __contains__(self, record: Record) -> bool:
        ''' Returns True if record was already processed. '''
        return record.path in self._processed
    def items(self) -> Iterable[Tuple[str, Dict[Record, List[Any]]]]:
        ''' Iterable with (group, {record: [extras]}) tuples. '''
        return self.state.items()
    def add(self, record: Record, sub_groups: Dict[str, List[Any]]) -> None:
        ''' Append groups if not processed already. {group: [extras]} '''
        if record.path not in self._processed:
            self._processed.add(record.path)
            for group, extras in sub_groups.items():
                if group in self.state:
                    self.state[group][record] = extras
                else:
                    self.state[group] = {record: extras}
 # -----------------------------------
 #              Watcher
 # -----------------------------------
 class Watcher:
    '''
    Callback is called with (Record, FieldKeyPath, field-value).
    Callback may yield one or more (group, extra-info) tuples.
    '''
    def __init__(self, config: Config) -> None:
        self.config = config
        self.flatten = True
        self.callback = None  # type: GroupingCallback #type:ignore[assignment]
    def grouping(self, flatten: bool = True) \
            -> Callable[[GroupingCallback], None]:
        '''
        Decorator to subscribe to attrib-elements.
        If flatten = False, dont explode FlowType.
        (record, field-key, field) -> (group, extra-info)
        '''
        def _decorator(fn: GroupingCallback) -> None:
            self.flatten = flatten
            self.callback = fn
        return _decorator
    def initialize(self, db: Database) -> None:
        ''' Reset internal state. You must initialize before each build! '''
        assert callable(self.callback), 'No grouping callback provided.'
        self._root = self.config.root
        self._state = GroupByState()
        self._model_reader = GroupByModelReader(db, attrib=self.config.key)
    def should_process(self, node: Record) -> bool:
        ''' Check if record path is being watched. '''
        return node['_path'].startswith(self._root)
    def process(self, record: Record) -> None:
        '''
        Will iterate over all record fields and call the callback method.
        Each record is guaranteed to be processed only once.
        '''
        if record in self._state:
            return
        tmp = {}  # type: Dict[str, List[Any]] # {group: [extras]}
        for key, field in self._model_reader.read(record, self.flatten):
            _gen = self.callback(GroupByCallbackArgs(record, key, field))
            try:
                obj = next(_gen)
                while True:
                    if not isinstance(obj, (str, tuple)):
                        raise TypeError(f'Unsupported groupby yield: {obj}')
                    group = obj if isinstance(obj, str) else obj[0]
                    if group not in tmp:
                        tmp[group] = []
                    if isinstance(obj, tuple):
                        tmp[group].append(obj[1])
                    # return slugified group key and continue iteration
                    if isinstance(_gen, Generator) and not _gen.gi_yieldfrom:
                        obj = _gen.send(self.config.slugify(group))
                    else:
                        obj = next(_gen)
            except StopIteration:
                del _gen
        self._state.add(record, tmp)
    def iter_sources(self, root: Record) -> Iterator[GroupBySource]:
        ''' Prepare and yield GroupBySource elements. '''
        for group, children in self._state.items():
            yield GroupBySource(root, group, self.config, children=children)
    def __repr__(self) -> str:
        return '<GroupByWatcher key="{}" enabled={} callback={}>'.format(
            self.config.key, self.config.enabled, self.callback)
--- a/setup.py
+++ b/setup.py
@@ -1,11 +1,7 @@
 from setuptools import setup
 import re
 with open('README.md') as fp:
    longdesc = fp.read()
    # replace fragment links with bold text
    frag_links = re.compile(r'\[([^]]+)\]\(#[^)]*\)')
    longdesc = frag_links.sub(lambda x: '__{}__'.format(x.group(1)), longdesc)
 setup(
    name='lektor-groupby',
@@ -17,7 +13,7 @@ setup(
    },
    author='relikd',
    url='https://github.com/relikd/lektor-groupby-plugin',
-    version='0.9',
+    version='0.9.3',
    description='Cluster arbitrary records with field attribute keyword.',
    long_description=longdesc,
    long_description_content_type="text/markdown",
Author	SHA1	Message	Date
relikd	298e0d4a62	v0.9.3	2022-04-06 15:47:38 +02:00
relikd	2a6bdf05fd	update example readme v0.9.3	2022-04-06 15:42:02 +02:00
relikd	df4be7c60a	builtin filter collision rename groupby -> vgroups	2022-04-06 13:29:19 +02:00
relikd	637524a615	update example to v0.9.3	2022-04-06 13:16:44 +02:00
relikd	a6d9f715f9	allow {key} in slug + allow sorting and hashing	2022-04-06 13:11:49 +02:00
relikd	d6df547682	config.root trailing slash + allow any in fields	2022-04-06 12:29:35 +02:00
relikd	ebc29459ec	remove ConfigKey and GroupKey types	2022-04-06 00:29:40 +02:00
relikd	adb26e343e	split py into modules	2022-04-05 22:58:53 +02:00
relikd	97b40b4886	refactoring II (watcher config + dependency mgmt)	2022-04-05 20:29:15 +02:00
relikd	479ff9b964	add virtual path resolver this allows the admin UI to preview groupby pages	2022-04-02 00:14:22 +02:00
relikd	626c0ab13a	fix processed lookup	2022-04-01 13:34:35 +02:00
relikd	2de02ed50c	add examples	2022-03-31 04:20:01 +02:00
relikd	c9c1ab69b1	complete refactoring to ensure group before build Due to a concurrency bug (lektor/lektor#1017), a source file is sporadically not updated because `before-build` is evaluated faster than `before-build-all`. Fixed with a redundant build process. Also: - adds before- and after-init hooks - encapsulates logic into separate classes - fix virtual path and remove virtual path resolver - more type hints (incl. bugfixes)	2022-03-31 04:16:18 +02:00