v0.9.7

- postpone building until really needed
- rebuild only if artifacts change
- no build on source update
- prune takes current resolver state instead of global var

.gitignore

@@ -1,5 +1,5 @@
 .DS_Store
-/dist-env/
+/env-publish/
 
 __pycache__/
 *.py[cod]

Makefile

@@ -1,21 +1,16 @@
-.PHONY: help
-help:
-	@echo 'commands:'
-	@echo '  dist'
-
-dist-env:
-	@echo Creating virtual environment...
-	@python3 -m venv 'dist-env'
-	@source dist-env/bin/activate && pip install twine
-
-.PHONY: dist
-dist: dist-env
-	[ -z "$${VIRTUAL_ENV}" ]  # you can not do this inside a virtual environment.
-	rm -rf dist
+dist: setup.py lektor_groupby/*
 	@echo Building...
 	python3 setup.py sdist bdist_wheel
-	@echo
 	rm -rf ./*.egg-info/ ./build/ MANIFEST
+
+env-publish:
+	@echo Creating virtual environment...
+	@python3 -m venv 'env-publish'
+	@source env-publish/bin/activate && pip install twine
+
+.PHONY: publish
+publish: dist env-publish
+	[ -z "$${VIRTUAL_ENV}" ]  # you can not do this inside a virtual environment.
 	@echo Publishing...
-	@echo "\033[0;31mEnter your PyPI token:\033[0m"
-	@source dist-env/bin/activate && export TWINE_USERNAME='__token__' && twine upload dist/*
+	@echo "\033[0;31mEnter PyPI token in password prompt:\033[0m"
+	@source env-publish/bin/activate && export TWINE_USERNAME='__token__' && twine upload dist/*

README.md

@@ -5,11 +5,10 @@ Can be used for tagging or similar tasks.
 The grouping algorithm is performed once.
 Contrary to, at least, cubic runtime if doing the same with Pad queries.
 
-To install this plugin, modify your Lektor project file:
+Install this plugin or modify your Lektor project file:
 
-```ini
-[packages]
-lektor-groupby = 0.9.1
+```sh
+lektor plugin add groupby
 ```
 
 Optionally, enable a basic config:

examples/Examples.lektorproject

@@ -2,4 +2,4 @@
 name = GroupBy Examples
 
 [packages]
-lektor-groupby = 0.9.1
+lektor-groupby = 0.9.7

examples/Makefile

@@ -0,0 +1,7 @@
+.PHONY: server clean plugins
+server:
+	lektor server
+clean:
+	lektor clean --yes -v
+plugins:
+	lektor plugins flush-cache && lektor plugins list

examples/README.md

@@ -1,16 +1,19 @@
 # Usage
 
 Overview:
-- the [quick config](#quick-config) example shows how you can use the plugin config to setup a quick and easy tagging system.
-- the [simple example](#simple-example) goes into detail how this plugin works.
-- the [advanced example](#advanced-example) touches on the potentials of the plugin.
+- [quick config example](#quick-config) shows how you can use the plugin config to setup a quick and easy tagging system.
+- [simple example](#simple-example) goes into detail how to use it in your own plugin.
+- [advanced example](#advanced-example) touches on the potentials of the plugin.
+- [Misc](#misc) shows other use-cases.
 
+After reading this tutorial, have a look at other plugins that use `lektor-groupby`:
+- [lektor-inlinetags](https://github.com/relikd/lektor-inlinetags-plugin)
 
 
 ## About
 
 To use the groupby plugin you have to add an attribute to your model file.
-In our case you can refer to the `models/page.ini` model:
+In our case you can refer to the [`models/page.ini`](./models/page.ini) model:
 
 ```ini
 [fields.tags]
@@ -35,19 +38,26 @@ The attribute name is later used for grouping.
 ## Quick config
 
 Relevant files:
-```
-configs/groupby.ini
-templates/example-config.html
-```
+
+- [`configs/groupby.ini`](./configs/groupby.ini)
+- [`templates/example-config.html`](./templates/example-config.html)
+
 
 The easiest way to add tags to your site is by defining the `groupby.ini` config file.
 
 ```ini
 [testA]
 root = /
-slug = config/{group}.html
+slug = config/{key}.html
 template = example-config.html
 split = ' '
+enabled = True
+
+[testA.fields]
+title = "Tagged: " ~ this.group
+
+[testA.key_map]
+Blog = News
 ```
 
 The configuration parameter are:
@@ -57,9 +67,11 @@ The configuration parameter are:
    All results will be placed under this directory, e.g., `/tags/tagname/`.
    If you use `root = /blog`, the results path will be `/blog/tags/tagname/`.
    The groupby plugin will traverse all sub-pages wich contain the attribute `testA`.
-3. The `slug` parameter (`config/{group}.html`) is where the results are placed.
+3. The `slug` parameter (`config/{key}.html`) is where the results are placed.
    In our case, the path resolves to `config/tagname.html`.
-   The default value is `{attrib}/{group}/index.html` which would resolve to `testA/tagname/index.html`.
+   The default value is `{attrib}/{key}/index.html` which would resolve to `testA/tagname/index.html`.
+   If this field contains `{key}`, it just replaces the value with the group-key.
+   In all other cases the field value is evaluated in a jinja context.
 4. The `template`parameter (`example-config.html`) is used to render the results page.
    If no explicit template is set, the default template `groupby-testA.html` will be used.
    Where `testA` is replaced with whatever attribute you chose.
@@ -68,18 +80,54 @@ The configuration parameter are:
    The split is only relevant for fields of type `string` or `text`.
    These single-line fields are then expanded to lists as well.
    If you do not provide the `split` option, the whole field value will be used as tagname.
+6. The `enabled` parameter allows you to quickly disable the grouping.
 
 
 You can have multiple listeners, e.g., one for `/blog/` and another for `/projects/`.
 Just create as many custom attributes as you like, each having its own section.
 
-In your template file you have access to the children (pages) and their tags.
-The emitted `extras` for the child is a list of original tagnames.
+There are two additional config mappings, `.fields` and `.key_map`.
+Key-value pairs in `.fields` will be added as attributes to your grouping.
+You can access them in your template (e.g., `{{this.title}}`).
+All of the `.fields` values are evaluted in a jinja context, so be cautious when using plain strings. 
+
+The built-in field attributes are:
+
+- `group`: returned group name, e.g., "A Title?"
+- `key`: slugified group value, e.g., "a-title"
+- `slug`: url path after root node, e.g. "config/a-title.html" (can be `None`)
+- `record`: parent node, e.g., `Page(path="/")`
+- `children`: dictionary of `{record: extras}` pairs
+- `first_child`: first page
+- `first_extra`: first extra
+- `config`: configuration object (see below)
+
+Without any changes, the `key` value will just be `slugify(group)`.
+However, the other mapping `.key_map` will replace `group` with whatever replacement value is provided in the `.key_map` and then slugified.
+You could, for example, add a `C# = c-sharp` mapping, which would otherwise just be slugified to `c`.
+This is equivalent to `slugify(key_map.get(group))`.
+
+The `config` attribute contains the values that created the group:
+
+- `key`: attribute key, e.g., `TestA`
+- `root`: as provided by init, e.g., `/`
+- `slug`: the raw value, e.g., `config/{key}.html`
+- `template`: as provided by init, e.g., `example-config.html`
+- `enabled`: boolean
+- `dependencies`: path to config file (if initialized from config)
+- `fields`: raw values from `TestA.fields`
+- `key_map`: raw values from `TestA.key_map`
+
+In your template file you have access to the attributes, config, and children (pages):
 
 ```jinja2
-{%- for child, extras in this.children.items() %}
-<li>Page: {{ child.path }}, Tags: {{ extras }}</li>
+<h2>{{ this.title }}</h2>
+<p>Key: {{ this.key }}, Attribute: {{ this.config.key }}</p>
+<ul>
+{%- for child in this.children %}
+<li>Page: {{ child.path }}</li>
 {%- endfor %}
+</ul>
 ```
 
 
@@ -87,66 +135,70 @@ The emitted `extras` for the child is a list of original tagnames.
 ## Simple example
 
 Relevant files:
-```
-packages/simple-example/lektor_simple.py
-templates/example-simple.html
-```
+
+- [`packages/simple-example/lektor_simple.py`](./packages/simple-example/lektor_simple.py)
+- [`templates/example-simple.html`](./templates/example-simple.html)
+
 
 ```python
-def on_groupby_after_build_all(self, groupby, builder, **extra):
-    @groupby.watch('/blog', 'testB', slug='simple/{group}/index.html',
-                   template='example-simple.html', flatten=True)
-    def convert_simple_example(args):
-        value = args.field  # list, since model is 'strings' type
-        for tag in value: 
-            yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
+def on_groupby_before_build_all(self, groupby, builder, **extra):
+    watcher = groupby.add_watcher('testB', {
+        'root': '/blog',
+        'slug': 'simple/{key}/index.html',
+        'template': 'example-simple.html',
+    })
+    watcher.config.set_key_map({'Foo': 'bar'})
+    watcher.config.set_fields({'date': datetime.now()})
 
-        # page = args.record  # extract additional info from source
-        # fieldKey, flowIndex, flowKey = args.key  # or get field index
-        # if flowIndex is None:
-        #     obj = page[fieldKey]
-        # else:
-        #     obj = page[fieldKey].blocks[flowIndex].get(flowKey)
+    @watcher.grouping(flatten=True)
+    def convert_simple_example(args):
+        # Yield groups
+        value = args.field  # type: list # since model is 'strings' type
+        for tag in value:
+            yield tag, {'tags_in_page': value}
 ```
 
-This example is roughly equivalent to the config file example.
-The parameters of the `@groupby.watch` function (`root`, `attribute`, `slug`, `template`) correspond to the same config parameters described above.
-There is a new `flatten` parameter:
+This example is roughly equivalent to the config example above – the parameters of the `groupby.add_watcher` function correspond to the same config parameters.
+Additionally, you can set other types in `set_fields` (all strings are evaluated in jinja context!).
 
-- Flatten determines how Flow elements are processed.
+`@watcher.grouping` sets the callback to generate group keys.
+It has one optional flatten parameter:
+
+- `flatten` determines how Flow elements are processed.
   If `False`, the callback function is called once per Flow element.
   If `True` (default), the callback is called for all Flow-Blocks of the Flow individually.
   The attribute `testB` can be attached to either the Flow or a Flow-Block regardless.
 
 The `args` parameter of the `convert_simple_example()` function is a named tuple, it has three attributes:
 
-1. The `record` points to the `Page` source which contains the tag.
+1. The `record` points to the `Page` record that contains the tag.
 2. The `key` tuple `(field-key, flow-index, flow-key)` tells which field is processed.
    For Flow types, `flow-index` and `flow-key` are set, otherwise they are `None`.
 3. The `field` value is the content of the processed field.
-   The field value is reoughly equivalent to the following:
+   The field value is equivalent to the following:
 
 ```python
-args.page[fieldKey].blocks[flowIndex].get(flowKey)
+k = args.key
+field = args.record[k.fieldKey].blocks[k.flowIndex].get(k.flowKey)
 ```
 
 The callback body **can** produce groupings but does not have to.
-If you choose to produce an entry, you have to `yield` a tuple pair of `(groupkey, extra-info)`.
-`groupkey` is used to combine & cluster pages and must be URL-safe.
-The `extra-info` is passed through to your template file.
-You can yield more than one entry per source or filter / ignore pages if you don't yield anything.
+If you choose to produce an entry, you have to `yield` a string or tuple pair `(group, extra-info)`.
+`group` is slugified (see above) and then used to combine & cluster pages.
+The `extra-info` (optional) is passed through to your template file.
+You can yield more than one entry per source.
+Or ignore pages if you don't yield anything.
 
 The template file can access and display the `extra-info`:
 
 ```jinja2
-{%- for child, extras in this.children.items() %}
-<b>Page: {{ child.title }}<b>
+<p>Custom field date: {{this.date}}</p>
 <ul>
-{%- for extra in extras %}
-<li>Name: {{ extra.val }}, Tag count: {{ extra.tags_in_page }}</li>
+{%- for child, extras in this.children.items() -%}
+{%- set etxra = (extras|first).tags_in_page %}
+<li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
 {%- endfor %}
 </ul>
-{%- endfor %}
 ```
 
 
@@ -154,11 +206,11 @@ The template file can access and display the `extra-info`:
 ## Advanced example
 
 Relevant files:
-```
-configs/advanced.ini
-packages/advanced-example/lektor_advanced.py
-templates/example-advanced.html
-```
+
+- [`configs/advanced.ini`](./configs/advanced.ini)
+- [`packages/advanced-example/lektor_advanced.py`](./packages/advanced-example/lektor_advanced.py)
+- [`templates/example-advanced.html`](./templates/example-advanced.html)
+
 
 The following example is similar to the previous one.
 Except that it loads a config file and replaces in-text occurrences of `{{Tagname}}` with `<a href="/tag/">Tagname</a>`.
@@ -166,44 +218,75 @@ Except that it loads a config file and replaces in-text occurrences of `{{Tagnam
 ```python
 def on_groupby_before_build_all(self, groupby, builder, **extra):
     # load config
-    regex = re.compile(self.get_config().get('match'))
-    # since we load and use a config file, we need to track the dependency
-    @groupby.depends_on(self.config_filename)
-    @groupby.watch('/', 'testC', slug='advanced/{group}/',
-                   template='example-advanced.html')
+    config = self.get_config()
+    regex = config.get('testC.pattern.match')
+    try:
+        regex = re.compile(regex)
+    except Exception as e:
+        print('inlinetags.regex not valid: ' + str(e))
+        return
+
+    # load config directly (which also tracks dependency)
+    watcher = groupby.add_watcher('testC', config)
+
+    @watcher.grouping()
     def convert_replace_example(args):
         # args.field assumed to be Markdown
         obj = args.field.source
+        slugify_map = {}  # type Dict[str, str]
         for match in regex.finditer(obj):
             tag = match.group(1)
-            yield slugify(tag), tag
+            key = yield tag
+            print('[advanced] slugify:', tag, '->', key)
+            slugify_map[tag] = key
 
         def _fn(match: re.Match) -> str:
             tag = match.group(1)
-            return f'<a href="/advanced/{slugify(tag)}/">{tag}</a>'
+            return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
         args.field.source = regex.sub(_fn, obj)
 ```
 
-One **important** thing to notice is, we use `on_groupby_before_build_all` to register our callback function.
-This is required because we would like to modify the source **before** it is written to disk.
-If you look back to the [simple example](#simple-example), we used `on_groupby_after_build_all` because we did not care when it is executed.
-Generally, it makes little difference which one you use (`on-after` is likely less busy).
-Just know that you can process the source before or after it is build.
+Notice, `add_watcher` accepts a config file as parameter which keeps also track of dependencies and rebuilds pages when you edit the config file.
+Further, the `yield` call returns the slugified group-key.
+First, you do not need to slugify it yourself and second, potential replacements from `key_map` are already handled.
 
 For Markdown fields, we can modify the `source` attribute directly.
-All other field typed need to be accessed via `args.record` key indirection.
+All other field types need to be accessed via `args.record` key indirection (see [simple example](#simple-example)).
 
 ```ini
+[testC]
+root = /
+slug = "advanced/{}/".format(this.key)
+template = example-advanced.html
+
+[testC.pattern]
 match = {{([^}]{1,32})}}
 ```
 
-Lastly, the config file contains a regular expression which matches `{{` + any string less than 32 characters + `}}`.
-Notice, the parenthesis (`()`) will match the inner part but the replace function (`re.sub`) will remove the `{{}}` too.
+The config file takes the same parameters as the [config example](#quick-config).
+As you can see, `slug` is evaluated in jinja context.
 
-If the user changes the regex pattern in the config file, we need to rebuild all tags.
-For this purpose we need to track changes to the config file.
-This is done by calling:
+We introduced a new config option `testC.pattern.match`.
+This regular expression matches `{{` + any string less than 32 characters + `}}`.
+Notice, the parenthesis (`()`) will match only the inner part but the replace function (`re.sub`) will remove the `{{}}`.
 
-```python
-@groupby.depends_on(file1, file2, ...)
+
+
+## Misc
+
+It was shortly mentioned above that slugs can be `None` (only if manually set to `slug = None`).
+This is useful if you do not want to create subpages but rather an index page containing all groups.
+This can be done in combination with the next use-case:
+
+```jinja2
+{%- for x in this|vgroups(keys=['TestA', 'TestB'], fields=[], flows=[], recursive=True)|unique|sort %} 
+<a href="{{ x|url }}">({{ x.group }})</a>
+{%- endfor %} 
 ```
+
+You can query the groups of any parent node (including those without slug).
+[`templates/page.html`](./templates/page.html) uses this.
+The keys (`'TestA', 'TestB'`) can be omitted which will return all groups of all attributes (you can still filter them with `x.config.key == 'TestC'`).
+The `fields` and `flows` params are also optional.
+With these you can match groups in `args.key.fieldKey` and `args.key.flowKey`.
+For example, if you have a “tags” field and an “additional-tags” field and you only want to show one in a preview.

examples/configs/advanced.ini

@@ -1 +1,15 @@
+[testC]
+root = /
+slug = "advanced/{}/".format(this.key)
+template = example-advanced.html
+
+[testC.pattern]
 match = {{([^}]{1,32})}}
+
+[testC.fields]
+desc = "Tag: " ~ this.group ~ ", Key: " ~ this.key
+
+[testC.key_map]
+Blog = case-sensitive
+Two = three
+three = no-nested-replace

examples/configs/groupby.ini

@@ -1,5 +1,12 @@
 [testA]
+enabled = True
 root = /
-slug = config/{group}.html
+slug = config/{key}.html
 template = example-config.html
 split = ' '
+
+[testA.fields]
+title = "Tagged: " ~ this.group
+
+[testA.key_map]
+Blog = News

examples/packages/advanced-example/lektor_advanced.py

@@ -1,31 +1,36 @@
 # -*- coding: utf-8 -*-
 from lektor.pluginsystem import Plugin
-from lektor.utils import slugify
+from typing import Generator
 import re
+from lektor_groupby import GroupBy, GroupByCallbackArgs
 
 
 class AdvancedGroupByPlugin(Plugin):
-    def on_groupby_before_build_all(self, groupby, builder, **extra):
+    def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
         # load config
-        regex = self.get_config().get('match')
+        config = self.get_config()
+        regex = config.get('testC.pattern.match')
         try:
             regex = re.compile(regex)
         except Exception as e:
             print('inlinetags.regex not valid: ' + str(e))
             return
 
-        # since we load and use a config file, we need to track the dependency
-        @groupby.depends_on(self.config_filename)
-        @groupby.watch('/', 'testC', slug='advanced/{group}/',
-                       template='example-advanced.html')
-        def convert_replace_example(args):
+        # load config directly (which also tracks dependency)
+        watcher = groupby.add_watcher('testC', config)
+
+        @watcher.grouping()
+        def _replace(args: GroupByCallbackArgs) -> Generator[str, str, None]:
             # args.field assumed to be Markdown
             obj = args.field.source
+            slugify_map = {}  # type Dict[str, str]
             for match in regex.finditer(obj):
                 tag = match.group(1)
-                yield slugify(tag), tag
+                key = yield tag
+                print('[advanced] slugify:', tag, '->', key)
+                slugify_map[tag] = key
 
             def _fn(match: re.Match) -> str:
                 tag = match.group(1)
-                return f'<a href="/advanced/{slugify(tag)}/">{tag}</a>'
+                return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
             args.field.source = regex.sub(_fn, obj)

examples/packages/simple-example/lektor_simple.py

@@ -1,17 +1,26 @@
 # -*- coding: utf-8 -*-
 from lektor.pluginsystem import Plugin
-from lektor.utils import slugify
+from typing import Iterator, Tuple
+from datetime import datetime
+from lektor_groupby import GroupBy, GroupByCallbackArgs
 
 
 class SimpleGroupByPlugin(Plugin):
-    def on_groupby_after_build_all(self, groupby, builder, **extra):
-        @groupby.watch('/blog', 'testB', slug='simple/{group}/index.html',
-                       template='example-simple.html', flatten=True)
-        def convert_simple_example(args):
+    def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
+        watcher = groupby.add_watcher('testB', {
+            'root': '/blog',
+            'slug': 'simple/{key}/index.html',
+            'template': 'example-simple.html',
+        })
+        watcher.config.set_key_map({'Foo': 'bar'})
+        watcher.config.set_fields({'date': datetime.now()})
+
+        @watcher.grouping(flatten=True)
+        def fn_simple(args: GroupByCallbackArgs) -> Iterator[Tuple[str, dict]]:
             # Yield groups
-            value = args.field  # list type since model is 'strings' type
+            value = args.field  # type: list # since model is 'strings' type
             for tag in value:
-                yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
+                yield tag, {'tags_in_page': value}
             # Everything below is just for documentation purposes
             page = args.record  # extract additional info from source
             fieldKey, flowIndex, flowKey = args.key  # or get field index
@@ -19,6 +28,6 @@ class SimpleGroupByPlugin(Plugin):
                 obj = page[fieldKey]
             else:
                 obj = page[fieldKey].blocks[flowIndex].get(flowKey)
-            print('page:', page)
-            print(' obj:', obj)
-            print()
+            print('[simple] page:', page)
+            print('[simple]  obj:', obj)
+            print('[simple] ')

examples/templates/example-advanced.html

@@ -1,3 +1,4 @@
 <h2>Path: {{ this | url(absolute=True) }}</h2>
-<div>This is: {{this}}</div>
-<div>Children: {{this.children}}</div>
+<p>This is: {{this}}</p>
+<p>Custom field, desc: "{{this.desc}}"</p>
+<p>Children: {{this.children}}</p>

examples/templates/example-config.html

@@ -1,7 +1,9 @@
 <h2>Path: {{ this | url(absolute=True) }}</h2>
-<div>This is: {{this}}</div>
+<p>This is: {{this}}</p>
+<p>Group: "{{this.group}}", Key: "{{this.key}}"</p>
+<p>Custom field title: {{this.title}}</p>
 <ul>
-	{%- for child, extras in this.children.items() %}
-	<li>Page: {{ child.path }}, Tags: {{ extras }}</li>
+{%- for child in this.children %}
+<li>Child: <a href="{{child|url}}">{{child.title}}</a> ({{child.path}})</li>
 {%- endfor %}
 </ul>

examples/templates/example-simple.html

@@ -1,12 +1,9 @@
 <h2>Path: {{ this | url(absolute=True) }}</h2>
-<div>This is: {{this}}</div>
+<p>This is: {{this}}</p>
+<p>Custom field date: {{this.date}}</p>
 <ul>
-	{%- for child, extras in this.children.items() %}
-	<li>Page: {{ child.path }}</li>
-	<ul>
-		{%- for extra in extras %}
-		Name: {{ extra.val }}, Tag count: {{ extra.tags_in_page }}
-		{%- endfor %}
-	</ul>
+{%- for child, extras in this.children.items() -%}
+{%- set etxra = (extras|first).tags_in_page %}
+<li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
 {%- endfor %}
 </ul>

examples/templates/page.html

@@ -18,20 +18,12 @@ main { margin: 3em; }
     {% block body %}{{ this.body }}{% endblock %}
   </main>
   <footer>
-    <div>Simple Tags:
-      {% for tag in ['blog','directory','blog-post','initial','samegroup'] %}
-      <a href="/blog/simple/{{tag}}/">({{tag}})</a>
-      {% endfor %}
-    </div>
-    <div>Config Tags:
-      {% for tag in ['root','blog','directory','blog-post','initial','samegroup'] %}
-      <a href="/config/{{tag}}.html">({{tag}})</a>
-      {% endfor %}
-    </div>
-    <div>Advanced Tags:
-      {% for tag in ['tag','two','blog'] %}
-      <a href="/advanced/{{tag}}/">({{tag}})</a>
-      {% endfor %}
+    {%- for k, v in [('testA','Config'),('testB','Simple'),('testC','Advanced')] %}
+    <div>{{v}} Tags:
+      {%- for x in this|vgroups(k, recursive=True)|unique|sort %} 
+        <a href="{{ x|url }}">({{x.key}})</a>
+      {%- endfor %} 
     </div>
+    {%- endfor %}
   </footer>
 </body>

lektor_groupby.py

@@ -1,498 +0,0 @@
-# -*- coding: utf-8 -*-
-from lektor.db import Database, Record  # typing
-from lektor.build_programs import BuildProgram
-from lektor.builder import Artifact, Builder  # typing
-from lektor.pluginsystem import Plugin
-from lektor.reporter import reporter
-from lektor.sourceobj import SourceObject, VirtualSourceObject
-from lektor.types.flow import Flow, FlowType
-from lektor.utils import bool_from_string, build_url, prune_file_and_folder
-# for quick config
-from lektor.utils import slugify
-
-from typing import Tuple, Dict, Set, List, NamedTuple
-from typing import NewType, Optional, Iterator, Callable, Iterable
-
-VPATH = '@groupby'  # potentially unsafe. All matching entries are pruned.
-
-
-# -----------------------------------
-#            Typing
-# -----------------------------------
-AttributeKey = NewType('AttributeKey', str)  # attribute of lektor model
-GroupKey = NewType('GroupKey', str)  # key of group-by
-
-
-class ResolverConf(NamedTuple):
-    attrib: AttributeKey
-    group: GroupKey
-    slug: str
-
-
-class FieldKeyPath(NamedTuple):
-    fieldKey: str
-    flowIndex: Optional[int] = None
-    flowKey: Optional[str] = None
-
-
-class GroupByCallbackArgs(NamedTuple):
-    record: Record
-    key: FieldKeyPath
-    field: object  # lektor model data-field value
-
-
-GroupingCallback = Callable[[GroupByCallbackArgs],
-                            Iterator[Tuple[GroupKey, object]]]
-
-
-# -----------------------------------
-#    VirtualSource & BuildProgram
-# -----------------------------------
-
-
-class GroupBySource(VirtualSourceObject):
-    '''
-    Holds information for a single group/cluster.
-    This object is accessible in your template file.
-    Attributes: record, attrib, group, slug, template, children
-
-    :DEFAULTS:
-    slug: "{attrib}/{group}/index.html"
-    template: "groupby-attribute.html"
-    '''
-
-    def __init__(
-        self,
-        record: Record,
-        attrib: AttributeKey,
-        group: GroupKey, *,
-        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
-        template: Optional[str] = None  # default: "groupby-attrib.html"
-    ) -> None:
-        super().__init__(record)
-        self.attrib = attrib
-        self.group = group
-        self.template = template or 'groupby-{}.html'.format(self.attrib)
-        # custom user path
-        slug = slug or '{attrib}/{group}/index.html'
-        slug = slug.replace('{attrib}', self.attrib)
-        slug = slug.replace('{group}', self.group)
-        if slug.endswith('/index.html'):
-            slug = slug[:-10]
-        self.slug = slug
-        # user adjustable after init
-        self.children = {}  # type: Dict[Record, List[object]]
-        self.dependencies = set()  # type: Set[str]
-
-    @property
-    def path(self) -> str:
-        # Used in VirtualSourceInfo, used to prune VirtualObjects
-        return f'{self.record.path}{VPATH}/{self.attrib}/{self.group}'
-
-    @property
-    def url_path(self) -> str:
-        # Actual path to resource as seen by the browser
-        return build_url([self.record.path, self.slug])
-
-    def iter_source_filenames(self) -> Iterator[str]:
-        ''' Enumerate all dependencies '''
-        if self.dependencies:
-            yield from self.dependencies
-        for record in self.children:
-            yield from record.iter_source_filenames()
-
-    def __str__(self) -> str:
-        txt = '<GroupBySource'
-        for x in ['attrib', 'group', 'slug', 'template']:
-            txt += ' {}="{}"'.format(x, getattr(self, x))
-        return txt + ' children={}>'.format(len(self.children))
-
-
-class GroupByBuildProgram(BuildProgram):
-    ''' Generate Build-Artifacts and write files. '''
-
-    def produce_artifacts(self) -> None:
-        url = self.source.url_path
-        if url.endswith('/'):
-            url += 'index.html'
-        self.declare_artifact(url, sources=list(
-            self.source.iter_source_filenames()))
-        GroupByPruner.track(url)
-
-    def build_artifact(self, artifact: Artifact) -> None:
-        self.source.pad.db.track_record_dependency(self.source)
-        artifact.render_template_into(self.source.template, this=self.source)
-
-
-# -----------------------------------
-#              Helper
-# -----------------------------------
-
-
-class GroupByPruner:
-    '''
-    Static collector for build-artifact urls.
-    All non-tracked VPATH-urls will be pruned after build.
-    '''
-    _cache: Set[str] = set()
-    # Note: this var is static or otherwise two instances of
-    #       GroupByCreator would prune each others artifacts.
-
-    @classmethod
-    def track(cls, url: str) -> None:
-        ''' Add url to build cache to prevent pruning. '''
-        cls._cache.add(url.lstrip('/'))
-
-    @classmethod
-    def prune(cls, builder: Builder) -> None:
-        ''' Remove previously generated, unreferenced Artifacts. '''
-        dest_path = builder.destination_path
-        con = builder.connect_to_database()
-        try:
-            with builder.new_build_state() as build_state:
-                for url, file in build_state.iter_artifacts():
-                    if url.lstrip('/') in cls._cache:
-                        continue  # generated in this build-run
-                    infos = build_state.get_artifact_dependency_infos(url, [])
-                    for v_path, _ in infos:
-                        if VPATH not in v_path:
-                            continue  # we only care about groupby Virtuals
-                        reporter.report_pruned_artifact(url)
-                        prune_file_and_folder(file.filename, dest_path)
-                        build_state.remove_artifact(url)
-                        break  # there is only one VPATH-entry per source
-        finally:
-            con.close()
-        cls._cache.clear()
-
-
-class GroupByModelReader:
-    ''' Find models and flow-models which contain attrib '''
-
-    def __init__(self, db: Database, attrib: AttributeKey) -> None:
-        self._flows = {}  # type: Dict[str, Set[str]]
-        self._models = {}  # type: Dict[str, Dict[str, str]]
-        # find flow blocks with attrib
-        for key, flow in db.flowblocks.items():
-            tmp1 = set(f.name for f in flow.fields
-                       if bool_from_string(f.options.get(attrib, False)))
-            if tmp1:
-                self._flows[key] = tmp1
-        # find models with attrib or flow-blocks containing attrib
-        for key, model in db.datamodels.items():
-            tmp2 = {}  # Dict[str, str]
-            for field in model.fields:
-                if bool_from_string(field.options.get(attrib, False)):
-                    tmp2[field.name] = '*'  # include all children
-                elif isinstance(field.type, FlowType) and self._flows:
-                    # only processed if at least one flow has attrib
-                    fbs = field.type.flow_blocks
-                    # if fbs == None, all flow-blocks are allowed
-                    if fbs is None or any(x in self._flows for x in fbs):
-                        tmp2[field.name] = '?'  # only some flow blocks
-            if tmp2:
-                self._models[key] = tmp2
-
-    def read(
-        self,
-        record: Record,
-        flatten: bool = False
-    ) -> Iterator[Tuple[FieldKeyPath, object]]:
-        '''
-        Enumerate all fields of a Record with attrib = True.
-        Flows are either returned directly (flatten=False) or
-        expanded so that each flow-block is yielded (flatten=True)
-        '''
-        assert isinstance(record, Record)
-        for r_key, subs in self._models.get(record.datamodel.id, {}).items():
-            if subs == '*':  # either normal field or flow type (all blocks)
-                field = record[r_key]
-                if flatten and isinstance(field, Flow):
-                    for i, flow in enumerate(field.blocks):
-                        flowtype = flow['_flowblock']
-                        for f_key, block in flow._data.items():
-                            if f_key.startswith('_'):  # e.g., _flowblock
-                                continue
-                            yield FieldKeyPath(r_key, i, f_key), block
-                else:
-                    yield FieldKeyPath(r_key), field
-            else:  # always flow type (only some blocks)
-                for i, flow in enumerate(record[r_key].blocks):
-                    flowtype = flow['_flowblock']
-                    for f_key in self._flows.get(flowtype, []):
-                        yield FieldKeyPath(r_key, i, f_key), flow[f_key]
-
-
-class GroupByState:
-    ''' Holds and updates a groupby build state. '''
-
-    def __init__(self) -> None:
-        self.state = {}  # type: Dict[GroupKey, Dict[Record, List]]
-        self._processed = set()  # type: Set[Record]
-
-    def __contains__(self, record: Record) -> bool:
-        ''' Returns True if record was already processed. '''
-        return record in self._processed
-
-    def items(self) -> Iterable[Tuple[GroupKey, Dict]]:
-        ''' Iterable with (GroupKey, {record: extras}) tuples. '''
-        return self.state.items()
-
-    def add(self, record: Record, group: Dict[GroupKey, List]) -> None:
-        ''' Append groups if not processed already. '''
-        if record not in self._processed:
-            self._processed.add(record)
-            for group_key, extras in group.items():
-                if group_key in self.state:
-                    self.state[group_key][record] = extras
-                else:
-                    self.state[group_key] = {record: extras}
-
-
-class GroupByWatcher:
-    '''
-    Callback is called with (Record, FieldKeyPath, field-value).
-    Callback may yield one or more (group-key, extra-info) tuples.
-    '''
-
-    def __init__(
-        self,
-        root: str,
-        attrib: AttributeKey,
-        callback: GroupingCallback, *,
-        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
-        template: Optional[str] = None  # default: "groupby-attrib.html"
-    ) -> None:
-        self.root = root
-        self.attrib = attrib
-        self.callback = callback
-        self.slug = slug
-        self.template = template
-        # user editable attributes
-        self.flatten = True  # if False, dont explode FlowType
-        self.dependencies = set()  # type: Set[str]
-
-    def initialize(self, db: Database) -> None:
-        ''' Reset internal state. You must initialize before each build! '''
-        self._state = GroupByState()
-        self._model_reader = GroupByModelReader(db, self.attrib)
-
-    def should_process(self, node: SourceObject) -> bool:
-        ''' Check if record path is being watched. '''
-        if isinstance(node, Record):
-            p = node['_path']  # type: str
-            return p.startswith(self.root) or p + '/' == self.root
-        return False
-
-    def process(self, record: Record) -> None:
-        '''
-        Will iterate over all record fields and call the callback method.
-        Each record is guaranteed to be processed only once.
-        '''
-        if record in self._state:
-            return
-        tmp = {}
-        for key, field in self._model_reader.read(record, self.flatten):
-            for ret in self.callback(GroupByCallbackArgs(record, key, field)):
-                assert isinstance(ret, (tuple, list)), \
-                    'Must return tuple (group-key, extra-info)'
-                group_key, extra = ret
-                if group_key not in tmp:
-                    tmp[group_key] = [extra]
-                else:
-                    tmp[group_key].append(extra)
-        self._state.add(record, tmp)
-
-    def iter_sources(self, root: Record) -> Iterator[GroupBySource]:
-        ''' Prepare and yield GroupBySource elements. '''
-        for group_key, children in self._state.items():
-            src = GroupBySource(root, self.attrib, group_key,
-                                slug=self.slug, template=self.template)
-            src.dependencies = self.dependencies
-            src.children = children
-            yield src
-
-    def __str__(self) -> str:
-        txt = '<GroupByWatcher'
-        for x in [
-            'root', 'attrib', 'slug', 'template', 'flatten', 'dependencies'
-        ]:
-            txt += ' {}="{}"'.format(x, getattr(self, x))
-        return txt + '>'
-
-
-# -----------------------------------
-#           Main Component
-# -----------------------------------
-
-
-class GroupByCreator:
-    '''
-    Process all children with matching conditions under specified page.
-    Creates a grouping of pages with similar (self-defined) attributes.
-    The grouping is performed only once per build.
-    '''
-
-    def __init__(self) -> None:
-        self._watcher = []  # type: List[GroupByWatcher]
-        self._results = {}  # type: Dict[str, GroupBySource]
-        self._resolve_map = {}  # type: Dict[str, ResolverConf]
-
-    # ----------------
-    #   Add Observer
-    # ----------------
-
-    def depends_on(self, *args: str) \
-            -> Callable[[GroupByWatcher], GroupByWatcher]:
-        ''' Set GroupBySource dependency, e.g., a plugin config file. '''
-        def _decorator(r: GroupByWatcher) -> GroupByWatcher:
-            r.dependencies.update(list(args))
-            return r
-        return _decorator
-
-    def watch(
-        self,
-        root: str,
-        attrib: AttributeKey, *,
-        slug: Optional[str] = None,  # default: "{attrib}/{group}/index.html"
-        template: Optional[str] = None,  # default: "groupby-attrib.html"
-        flatten: bool = True,  # if False, dont explode FlowType
-    ) -> Callable[[GroupingCallback], GroupByWatcher]:
-        '''
-        Decorator to subscribe to attrib-elements.
-        (record, field-key, field) -> (group-key, extra-info)
-
-        :DEFAULTS:
-        slug: "{attrib}/{group}/index.html"
-        template: "groupby-attrib.html"
-        '''
-        root = root.rstrip('/') + '/'
-
-        def _decorator(fn: GroupingCallback) -> GroupByWatcher:
-            w = GroupByWatcher(root, attrib, fn, slug=slug, template=template)
-            w.flatten = flatten
-            self._watcher.append(w)
-            return w
-
-        return _decorator
-
-    # -----------
-    #   Builder
-    # -----------
-
-    def clear_previous_results(self) -> None:
-        ''' Reset prvious results. Must be called before each build. '''
-        self._watcher.clear()
-        self._results.clear()
-        self._resolve_map.clear()
-
-    def make_cluster(self, builder: Builder) -> None:
-        ''' Perform groupby, iterate over all children. '''
-        if not self._watcher:
-            return
-        for w in self._watcher:
-            w.initialize(builder.pad.db)
-
-        queue = builder.pad.get_all_roots()  # type: List[SourceObject]
-        while queue:
-            record = queue.pop()
-            self.queue_now(record)
-            if hasattr(record, 'attachments'):
-                queue.extend(record.attachments)  # type: ignore[attr-defined]
-            if hasattr(record, 'children'):
-                queue.extend(record.children)  # type: ignore[attr-defined]
-        # build artifacts
-        for w in self._watcher:
-            root = builder.pad.get(w.root)
-            for vobj in w.iter_sources(root):
-                self._results[vobj.url_path] = vobj
-        self._watcher.clear()
-
-    def queue_now(self, node: SourceObject) -> None:
-        ''' Process record immediatelly (No-Op if already processed). '''
-        for w in self._watcher:
-            if w.should_process(node):  # ensures type Record
-                w.process(node)  # type: ignore[arg-type]
-
-    def build_all(self, builder: Builder) -> None:
-        ''' Create virtual objects and build sources. '''
-        for url, x in sorted(self._results.items()):
-            builder.build(x)
-            self._resolve_map[url] = ResolverConf(x.attrib, x.group, x.slug)
-        self._results.clear()
-
-    # -----------------
-    #   Path resolver
-    # -----------------
-
-    def resolve_dev_server_path(
-        self, node: SourceObject, pieces: List[str]
-    ) -> Optional[GroupBySource]:
-        ''' Dev server only: Resolves path/ -> path/index.html '''
-        if not isinstance(node, Record):
-            return None
-        conf = self._resolve_map.get(build_url([node.url_path] + pieces))
-        if not conf:
-            return None
-        return GroupBySource(node, conf.attrib, conf.group, slug=conf.slug)
-
-
-# -----------------------------------
-#           Plugin Entry
-# -----------------------------------
-
-
-class GroupByPlugin(Plugin):
-    name = 'GroupBy Plugin'
-    description = 'Cluster arbitrary records with field attribute keyword.'
-
-    def on_setup_env(self, **extra: object) -> None:
-        self.creator = GroupByCreator()
-        self.env.add_build_program(GroupBySource, GroupByBuildProgram)
-
-        # resolve /tag/rss/ -> /tag/rss/index.html (local server only)
-        @self.env.urlresolver
-        def _(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
-            return self.creator.resolve_dev_server_path(node, parts)
-
-    def _load_quick_config(self) -> None:
-        ''' Load config file quick listeners. '''
-        config = self.get_config()
-        for attrib in config.sections():
-            sect = config.section_as_dict(attrib)
-            root = sect.get('root', '/')
-            slug = sect.get('slug')
-            temp = sect.get('template')
-            split = sect.get('split')
-
-            @self.creator.depends_on(self.config_filename)
-            @self.creator.watch(root, attrib, slug=slug, template=temp)
-            def _fn(args: GroupByCallbackArgs) \
-                    -> Iterator[Tuple[GroupKey, object]]:
-                val = args.field
-                if isinstance(val, str):
-                    val = val.split(split) if split else [val]  # make list
-                if isinstance(val, list):
-                    for tag in val:
-                        yield slugify(tag), tag
-
-    def on_before_build_all(self, builder: Builder, **extra: object) -> None:
-        self.creator.clear_previous_results()
-        # let other plugins register their @groupby.watch functions
-        self.emit('before-build-all', groupby=self.creator, builder=builder)
-        self.creator.make_cluster(builder)
-
-    def on_before_build(self, source: SourceObject, **extra: object) -> None:
-        # before-build may be called before before-build-all (issue #1017)
-        # make sure it is evaluated immediatelly
-        self.creator.queue_now(source)
-
-    def on_after_build_all(self, builder: Builder, **extra: object) -> None:
-        self.emit('after-build-all', groupby=self.creator, builder=builder)
-        self._load_quick_config()
-        self.creator.make_cluster(builder)
-        self.creator.build_all(builder)
-
-    def on_after_prune(self, builder: Builder, **extra: object) -> None:
-        # TODO: find a better way to prune unreferenced elements
-        GroupByPruner.prune(builder)

lektor_groupby/__init__.py

@@ -0,0 +1,4 @@
+from .config import Config  # noqa: F401
+from .groupby import GroupBy  # noqa: F401
+from .plugin import GroupByPlugin  # noqa: F401
+from .watcher import GroupByCallbackArgs  # noqa: F401

lektor_groupby/backref.py

@@ -0,0 +1,83 @@
+from lektor.context import get_ctx
+from typing import TYPE_CHECKING, Union, Iterable, Iterator
+import weakref
+if TYPE_CHECKING:
+    from lektor.builder import Builder
+    from lektor.db import Record
+    from .groupby import GroupBy
+    from .model import FieldKeyPath
+    from .vobj import GroupBySource
+
+
+class WeakVGroupsList(list):
+    def add(self, strong: 'FieldKeyPath', weak: 'GroupBySource') -> None:
+        super().append((strong, weakref.ref(weak)))
+
+
+class GroupByRef:
+    @staticmethod
+    def of(builder: 'Builder') -> 'GroupBy':
+        ''' Get the GroupBy object of a builder. '''
+        return builder.__groupby  # type:ignore[attr-defined,no-any-return]
+
+    @staticmethod
+    def set(builder: 'Builder', groupby: 'GroupBy') -> None:
+        ''' Set the GroupBy object of a builder. '''
+        builder.__groupby = groupby  # type: ignore[attr-defined]
+
+
+class VGroups:
+    @staticmethod
+    def of(record: 'Record') -> WeakVGroupsList:
+        '''
+        Return the (weak) set of virtual objects of a page.
+        Creates a new set if it does not exist yet.
+        '''
+        try:
+            wset = record.__vgroups  # type: ignore[attr-defined]
+        except AttributeError:
+            wset = WeakVGroupsList()
+            record.__vgroups = wset  # type: ignore[attr-defined]
+        return wset  # type: ignore[no-any-return]
+
+    @staticmethod
+    def iter(
+        record: 'Record',
+        keys: Union[str, Iterable[str], None] = None,
+        *,
+        fields: Union[str, Iterable[str], None] = None,
+        flows: Union[str, Iterable[str], None] = None,
+        recursive: bool = False
+    ) -> Iterator['GroupBySource']:
+        ''' Extract all referencing groupby virtual objects from a page. '''
+        ctx = get_ctx()
+        if not ctx:
+            raise NotImplementedError("Shouldn't happen, where is my context?")
+        # get GroupBy object
+        builder = ctx.build_state.builder
+        groupby = GroupByRef.of(builder)
+        groupby.make_once(builder)  # ensure did cluster before
+        # manage config dependencies
+        for dep in groupby.dependencies:
+            ctx.record_dependency(dep)
+        # prepare filter
+        if isinstance(keys, str):
+            keys = [keys]
+        if isinstance(fields, str):
+            fields = [fields]
+        if isinstance(flows, str):
+            flows = [flows]
+        # find groups
+        proc_list = [record]
+        while proc_list:
+            page = proc_list.pop(0)
+            if recursive and hasattr(page, 'children'):
+                proc_list.extend(page.children)  # type: ignore[attr-defined]
+            for key, vobj in VGroups.of(page):
+                if fields and key.fieldKey not in fields:
+                    continue
+                if flows and key.flowKey not in flows:
+                    continue
+                if keys and vobj().config.key not in keys:
+                    continue
+                yield vobj()

lektor_groupby/config.py

@@ -0,0 +1,87 @@
+from inifile import IniFile
+from lektor.utils import slugify
+
+from typing import Set, Dict, Optional, Union, Any
+
+AnyConfig = Union['Config', IniFile, Dict]
+
+
+class Config:
+    '''
+    Holds information for GroupByWatcher and GroupBySource.
+    This object is accessible in your template file ({{this.config}}).
+
+    Available attributes:
+    key, root, slug, template, enabled, dependencies, fields, key_map
+    '''
+
+    def __init__(
+        self,
+        key: str, *,
+        root: Optional[str] = None,  # default: "/"
+        slug: Optional[str] = None,  # default: "{attr}/{group}/index.html"
+        template: Optional[str] = None,  # default: "groupby-{attr}.html"
+    ) -> None:
+        self.key = key
+        self.root = (root or '/').rstrip('/') or '/'
+        self.slug = slug or (key + '/{key}/')  # key = GroupBySource.key
+        self.template = template or f'groupby-{self.key}.html'
+        # editable after init
+        self.enabled = True
+        self.dependencies = set()  # type: Set[str]
+        self.fields = {}  # type: Dict[str, Any]
+        self.key_map = {}  # type: Dict[str, str]
+
+    def slugify(self, k: str) -> str:
+        ''' key_map replace and slugify. '''
+        return slugify(self.key_map.get(k, k))  # type: ignore[no-any-return]
+
+    def set_fields(self, fields: Optional[Dict[str, Any]]) -> None:
+        '''
+        The fields dict is a mapping of attrib = Expression values.
+        Each dict key will be added to the GroupBySource virtual object.
+        Each dict value is passed through jinja context first.
+        '''
+        self.fields = fields or {}
+
+    def set_key_map(self, key_map: Optional[Dict[str, str]]) -> None:
+        ''' This mapping replaces group keys before slugify. '''
+        self.key_map = key_map or {}
+
+    def __repr__(self) -> str:
+        txt = '<GroupByConfig'
+        for x in ['key', 'root', 'slug', 'template', 'enabled']:
+            txt += ' {}="{}"'.format(x, getattr(self, x))
+        txt += f' fields="{", ".join(self.fields)}"'
+        return txt + '>'
+
+    @staticmethod
+    def from_dict(key: str, cfg: Dict[str, str]) -> 'Config':
+        ''' Set config fields manually. Allowed: key, root, slug, template. '''
+        return Config(
+            key=key,
+            root=cfg.get('root'),
+            slug=cfg.get('slug'),
+            template=cfg.get('template'),
+        )
+
+    @staticmethod
+    def from_ini(key: str, ini: IniFile) -> 'Config':
+        ''' Read and parse ini file. Also adds dependency tracking. '''
+        cfg = ini.section_as_dict(key)  # type: Dict[str, str]
+        conf = Config.from_dict(key, cfg)
+        conf.enabled = ini.get_bool(key + '.enabled', True)
+        conf.dependencies.add(ini.filename)
+        conf.set_fields(ini.section_as_dict(key + '.fields'))
+        conf.set_key_map(ini.section_as_dict(key + '.key_map'))
+        return conf
+
+    @staticmethod
+    def from_any(key: str, config: AnyConfig) -> 'Config':
+        assert isinstance(config, (Config, IniFile, Dict))
+        if isinstance(config, Config):
+            return config
+        elif isinstance(config, IniFile):
+            return Config.from_ini(key, config)
+        elif isinstance(config, Dict):
+            return Config.from_dict(key, config)

lektor_groupby/groupby.py

@@ -0,0 +1,80 @@
+from lektor.builder import PathCache
+from lektor.db import Record  # isinstance
+from typing import TYPE_CHECKING, Set, List
+from .config import Config
+from .watcher import Watcher
+if TYPE_CHECKING:
+    from .config import AnyConfig
+    from lektor.builder import Builder
+    from lektor.sourceobj import SourceObject
+    from .resolver import Resolver
+    from .vobj import GroupBySource
+
+
+class GroupBy:
+    '''
+    Process all children with matching conditions under specified page.
+    Creates a grouping of pages with similar (self-defined) attributes.
+    The grouping is performed only once per build.
+    '''
+
+    def __init__(self, resolver: 'Resolver') -> None:
+        self._watcher = []  # type: List[Watcher]
+        self._results = []  # type: List[GroupBySource]
+        self.resolver = resolver
+
+    def add_watcher(self, key: str, config: 'AnyConfig') -> Watcher:
+        ''' Init Config and add to watch list. '''
+        w = Watcher(Config.from_any(key, config))
+        self._watcher.append(w)
+        return w
+
+    def get_dependencies(self) -> Set[str]:
+        deps = set()  # type: Set[str]
+        for w in self._watcher:
+            deps.update(w.config.dependencies)
+        return deps
+
+    def queue_all(self, builder: 'Builder') -> None:
+        ''' Iterate full site-tree and queue all children. '''
+        self.dependencies = self.get_dependencies()
+        # remove disabled watchers
+        self._watcher = [w for w in self._watcher if w.config.enabled]
+        if not self._watcher:
+            return
+        # initialize remaining (enabled) watchers
+        for w in self._watcher:
+            w.initialize(builder.pad)
+        # iterate over whole build tree
+        queue = builder.pad.get_all_roots()  # type: List[SourceObject]
+        while queue:
+            record = queue.pop()
+            if hasattr(record, 'attachments'):
+                queue.extend(record.attachments)  # type: ignore[attr-defined]
+            if hasattr(record, 'children'):
+                queue.extend(record.children)  # type: ignore[attr-defined]
+            if isinstance(record, Record):
+                for w in self._watcher:
+                    if w.should_process(record):
+                        w.process(record)
+
+    def make_once(self, builder: 'Builder') -> None:
+        ''' Perform groupby, iter over sources with watcher callback. '''
+        if self._watcher:
+            self.resolver.reset()
+            for w in self._watcher:
+                root = builder.pad.get(w.config.root)
+                for vobj in w.iter_sources(root):
+                    self._results.append(vobj)
+                    self.resolver.add(vobj)
+            self._watcher.clear()
+
+    def build_all(self, builder: 'Builder') -> None:
+        ''' Create virtual objects and build sources. '''
+        self.make_once(builder)  # in case no page used the |vgroups filter
+        path_cache = PathCache(builder.env)
+        for vobj in self._results:
+            if vobj.slug:
+                builder.build(vobj, path_cache)
+        del path_cache
+        self._results.clear()  # garbage collect weak refs

lektor_groupby/model.py

@@ -0,0 +1,66 @@
+from lektor.db import Database, Record  # typing
+from lektor.types.flow import Flow, FlowType
+from lektor.utils import bool_from_string
+from typing import Set, Dict, Tuple, Any, NamedTuple, Optional, Iterator
+
+
+class FieldKeyPath(NamedTuple):
+    fieldKey: str
+    flowIndex: Optional[int] = None
+    flowKey: Optional[str] = None
+
+
+class ModelReader:
+    '''
+    Find models and flow-models which contain attribute.
+    Flows are either returned directly (flatten=False) or
+    expanded so that each flow-block is yielded (flatten=True)
+    '''
+
+    def __init__(self, db: Database, attr: str, flatten: bool = False) -> None:
+        self.flatten = flatten
+        self._flows = {}  # type: Dict[str, Set[str]]
+        self._models = {}  # type: Dict[str, Dict[str, str]]
+        # find flow blocks containing attribute
+        for key, flow in db.flowblocks.items():
+            tmp1 = set(f.name for f in flow.fields
+                       if bool_from_string(f.options.get(attr, False)))
+            if tmp1:
+                self._flows[key] = tmp1
+        # find models and flow-blocks containing attribute
+        for key, model in db.datamodels.items():
+            tmp2 = {}  # Dict[str, str]
+            for field in model.fields:
+                if bool_from_string(field.options.get(attr, False)):
+                    tmp2[field.name] = '*'  # include all children
+                elif isinstance(field.type, FlowType) and self._flows:
+                    # only processed if at least one flow has attr
+                    fbs = field.type.flow_blocks
+                    # if fbs == None, all flow-blocks are allowed
+                    if fbs is None or any(x in self._flows for x in fbs):
+                        tmp2[field.name] = '?'  # only some flow blocks
+            if tmp2:
+                self._models[key] = tmp2
+
+    def read(self, record: Record) -> Iterator[Tuple[FieldKeyPath, Any]]:
+        ''' Enumerate all fields of a Record with attrib = True. '''
+        assert isinstance(record, Record)
+        for r_key, subs in self._models.get(record.datamodel.id, {}).items():
+            field = record[r_key]
+            if not field:
+                continue
+            if subs == '*':  # either normal field or flow type (all blocks)
+                if self.flatten and isinstance(field, Flow):
+                    for i, flow in enumerate(field.blocks):
+                        flowtype = flow['_flowblock']
+                        for f_key, block in flow._data.items():
+                            if f_key.startswith('_'):  # e.g., _flowblock
+                                continue
+                            yield FieldKeyPath(r_key, i, f_key), block
+                else:
+                    yield FieldKeyPath(r_key), field
+            else:  # always flow type (only some blocks)
+                for i, flow in enumerate(field.blocks):
+                    flowtype = flow['_flowblock']
+                    for f_key in self._flows.get(flowtype, []):
+                        yield FieldKeyPath(r_key, i, f_key), flow[f_key]

lektor_groupby/plugin.py

@@ -0,0 +1,81 @@
+from lektor.db import Page  # isinstance
+from lektor.pluginsystem import Plugin  # subclass
+from typing import TYPE_CHECKING, Iterator, Any
+from .backref import GroupByRef, VGroups
+from .groupby import GroupBy
+from .pruner import prune
+from .resolver import Resolver
+from .vobj import VPATH, GroupBySource, GroupByBuildProgram
+if TYPE_CHECKING:
+    from lektor.builder import Builder, BuildState
+    from lektor.sourceobj import SourceObject
+    from .watcher import GroupByCallbackArgs
+
+
+class GroupByPlugin(Plugin):
+    name = 'GroupBy Plugin'
+    description = 'Cluster arbitrary records with field attribute keyword.'
+
+    def on_setup_env(self, **extra: Any) -> None:
+        self.has_changes = False
+        self.resolver = Resolver(self.env)
+        self.env.add_build_program(GroupBySource, GroupByBuildProgram)
+        self.env.jinja_env.filters.update(vgroups=VGroups.iter)
+
+    def on_before_build(
+        self, builder: 'Builder', source: 'SourceObject', **extra: Any
+    ) -> None:
+        # before-build may be called before before-build-all (issue #1017)
+        # make sure it is always evaluated first
+        if isinstance(source, Page):
+            self._init_once(builder)
+
+    def on_after_build(self, build_state: 'BuildState', **extra: Any) -> None:
+        if build_state.updated_artifacts:
+            self.has_changes = True
+
+    def on_after_build_all(self, builder: 'Builder', **extra: Any) -> None:
+        # only rebuild if has changes (bypass idle builds)
+        # or the very first time after startup (url resolver & pruning)
+        if self.has_changes or not self.resolver.has_any:
+            self._init_once(builder).build_all(builder)  # updates resolver
+            self.has_changes = False
+
+    def on_after_prune(self, builder: 'Builder', **extra: Any) -> None:
+        # TODO: find a better way to prune unreferenced elements
+        prune(builder, VPATH, self.resolver.files)
+
+    # ------------
+    #   internal
+    # ------------
+
+    def _init_once(self, builder: 'Builder') -> GroupBy:
+        try:
+            return GroupByRef.of(builder)
+        except AttributeError:
+            groupby = GroupBy(self.resolver)
+            GroupByRef.set(builder, groupby)
+
+        self._load_quick_config(groupby)
+        # let other plugins register their @groupby.watch functions
+        self.emit('before-build-all', groupby=groupby, builder=builder)
+        groupby.queue_all(builder)
+        return groupby
+
+    def _load_quick_config(self, groupby: GroupBy) -> None:
+        ''' Load config file quick listeners. '''
+        config = self.get_config()
+        for key in config.sections():
+            if '.' in key:  # e.g., key.fields and key.key_map
+                continue
+
+            watcher = groupby.add_watcher(key, config)
+            split = config.get(key + '.split')  # type: str
+
+            @watcher.grouping()
+            def _fn(args: 'GroupByCallbackArgs') -> Iterator[str]:
+                val = args.field
+                if isinstance(val, str):
+                    val = map(str.strip, val.split(split)) if split else [val]
+                if isinstance(val, (list, map)):
+                    yield from val

lektor_groupby/pruner.py

@@ -0,0 +1,45 @@
+'''
+Static collector for build-artifact urls.
+All non-tracked VPATH-urls will be pruned after build.
+'''
+from lektor.reporter import reporter  # report_pruned_artifact
+from lektor.utils import prune_file_and_folder
+from typing import TYPE_CHECKING, Set, Iterable
+if TYPE_CHECKING:
+    from lektor.builder import Builder
+
+
+def _normalize_url_cache(url_cache: Iterable[str]) -> Set[str]:
+    cache = set()
+    for url in url_cache:
+        if url.endswith('/'):
+            url += 'index.html'
+        cache.add(url.lstrip('/'))
+    return cache
+
+
+def prune(builder: 'Builder', vpath: str, url_cache: Iterable[str]) -> None:
+    '''
+    Remove previously generated, unreferenced Artifacts.
+    All urls in url_cache must have a trailing "/index.html" (instead of "/")
+    and also, no leading slash, "blog/index.html" instead of "/blog/index.html"
+    '''
+    vpath = '@' + vpath.lstrip('@')  # just in case of user error
+    dest_path = builder.destination_path
+    url_cache = _normalize_url_cache(url_cache)
+    con = builder.connect_to_database()
+    try:
+        with builder.new_build_state() as build_state:
+            for url, file in build_state.iter_artifacts():
+                if url.lstrip('/') in url_cache:
+                    continue  # generated in this build-run
+                infos = build_state.get_artifact_dependency_infos(url, [])
+                for artifact_name, _ in infos:
+                    if vpath not in artifact_name:
+                        continue  # we only care about our Virtuals
+                    reporter.report_pruned_artifact(url)
+                    prune_file_and_folder(file.filename, dest_path)
+                    build_state.remove_artifact(url)
+                    break  # there is only one VPATH-entry per source
+    finally:
+        con.close()

lektor_groupby/resolver.py

@@ -0,0 +1,61 @@
+from lektor.db import Record  # isinstance
+from lektor.utils import build_url
+from typing import TYPE_CHECKING, Dict, List, Tuple, Optional, Iterable
+from .vobj import VPATH, GroupBySource
+if TYPE_CHECKING:
+    from lektor.environment import Environment
+    from lektor.sourceobj import SourceObject
+    from .config import Config
+
+
+class Resolver:
+    '''
+    Resolve virtual paths and urls ending in /.
+    Init will subscribe to @urlresolver and @virtualpathresolver.
+    '''
+
+    def __init__(self, env: 'Environment') -> None:
+        self._data = {}  # type: Dict[str, Tuple[str, str, Config]]
+        env.urlresolver(self.resolve_server_path)
+        env.virtualpathresolver(VPATH.lstrip('@'))(self.resolve_virtual_path)
+
+    @property
+    def has_any(self) -> bool:
+        return bool(self._data)
+
+    @property
+    def files(self) -> Iterable[str]:
+        return self._data
+
+    def reset(self) -> None:
+        ''' Clear previously recorded virtual objects. '''
+        self._data.clear()
+
+    def add(self, vobj: GroupBySource) -> None:
+        ''' Track new virtual object (only if slug is set). '''
+        if vobj.slug:
+            self._data[vobj.url_path] = (vobj.key, vobj.group, vobj.config)
+
+    # ------------
+    #   Resolver
+    # ------------
+
+    def resolve_server_path(self, node: 'SourceObject', pieces: List[str]) \
+            -> Optional[GroupBySource]:
+        ''' Local server only: resolve /tag/rss/ -> /tag/rss/index.html '''
+        if isinstance(node, Record):
+            rv = self._data.get(build_url([node.url_path] + pieces))
+            if rv:
+                return GroupBySource(node, rv[0]).finalize(rv[2], rv[1])
+        return None
+
+    def resolve_virtual_path(self, node: 'SourceObject', pieces: List[str]) \
+            -> Optional[GroupBySource]:
+        ''' Admin UI only: Prevent server error and null-redirect. '''
+        if isinstance(node, Record) and len(pieces) >= 2:
+            path = node['_path']  # type: str
+            attr, grp, *_ = pieces
+            for slug, group, conf in self._data.values():
+                if attr == conf.key and slug == grp and path == conf.root:
+                    return GroupBySource(node, slug).finalize(conf, group)
+        return None

lektor_groupby/util.py

@@ -0,0 +1,28 @@
+from lektor.reporter import reporter, style
+
+from typing import List, Dict
+
+
+def report_config_error(key: str, field: str, val: str, e: Exception) -> None:
+    ''' Send error message to Lektor reporter. Indicate which field is bad. '''
+    msg = '[ERROR] invalid config for [{}.{}] = "{}",  Error: {}'.format(
+        key, field, val, repr(e))
+    try:
+        reporter._write_line(style(msg, fg='red'))
+    except Exception:
+        print(msg)  # fallback in case Lektor API changes
+
+
+def most_used_key(keys: List[str]) -> str:
+    if len(keys) < 3:
+        return keys[0]  # TODO: first vs last occurrence
+    best_count = 0
+    best_key = ''
+    tmp = {}  # type: Dict[str, int]
+    for k in keys:
+        num = (tmp[k] + 1) if k in tmp else 1
+        tmp[k] = num
+        if num > best_count:  # TODO: (>) vs (>=), first vs last occurrence
+            best_count = num
+            best_key = k
+    return best_key

lektor_groupby/vobj.py

@@ -0,0 +1,163 @@
+from lektor.build_programs import BuildProgram  # subclass
+from lektor.context import get_ctx
+from lektor.environment import Expression
+from lektor.sourceobj import VirtualSourceObject  # subclass
+from lektor.utils import build_url
+from typing import TYPE_CHECKING, Dict, List, Any, Optional, Iterator
+from .util import report_config_error, most_used_key
+if TYPE_CHECKING:
+    from lektor.builder import Artifact
+    from lektor.db import Record
+    from .config import Config
+
+VPATH = '@groupby'  # potentially unsafe. All matching entries are pruned.
+
+
+# -----------------------------------
+#           VirtualSource
+# -----------------------------------
+
+class GroupBySource(VirtualSourceObject):
+    '''
+    Holds information for a single group/cluster.
+    This object is accessible in your template file.
+    Attributes: record, key, group, slug, children, config
+    '''
+
+    def __init__(self, record: 'Record', slug: str) -> None:
+        super().__init__(record)
+        self.key = slug
+        self._group_map = []  # type: List[str]
+        self._children = {}  # type: Dict[Record, List[Any]]
+
+    def append_child(self, child: 'Record', extra: Any, group: str) -> None:
+        if child not in self._children:
+            self._children[child] = [extra]
+        else:
+            self._children[child].append(extra)
+        # _group_map is later used to find most used group
+        self._group_map.append(group)
+
+    # -------------------------
+    #   Evaluate Extra Fields
+    # -------------------------
+
+    def finalize(self, config: 'Config', group: Optional[str] = None) \
+            -> 'GroupBySource':
+        self.config = config
+        self.group = group or most_used_key(self._group_map)
+        del self._group_map
+        # evaluate slug Expression
+        if config.slug and '{key}' in config.slug:
+            self.slug = config.slug.replace('{key}', self.key)
+        else:
+            self.slug = self._eval(config.slug, field='slug')
+            assert self.slug != Ellipsis, 'invalid config: ' + config.slug
+        if self.slug and self.slug.endswith('/index.html'):
+            self.slug = self.slug[:-10]
+        # extra fields
+        for attr, expr in config.fields.items():
+            setattr(self, attr, self._eval(expr, field='fields.' + attr))
+        return self
+
+    def _eval(self, value: Any, *, field: str) -> Any:
+        ''' Internal only: evaluates Lektor config file field expression. '''
+        if not isinstance(value, str):
+            return value
+        pad = self.record.pad
+        alt = self.record.alt
+        try:
+            return Expression(pad.env, value).evaluate(pad, this=self, alt=alt)
+        except Exception as e:
+            report_config_error(self.config.key, field, value, e)
+            return Ellipsis
+
+    # ---------------------
+    #   Lektor properties
+    # ---------------------
+
+    @property
+    def path(self) -> str:
+        # Used in VirtualSourceInfo, used to prune VirtualObjects
+        return f'{self.record.path}{VPATH}/{self.config.key}/{self.key}'
+
+    @property
+    def url_path(self) -> str:
+        # Actual path to resource as seen by the browser
+        return build_url([self.record.path, self.slug])  # slug can be None!
+
+    def iter_source_filenames(self) -> Iterator[str]:
+        ''' Enumerate all dependencies '''
+        if self.config.dependencies:
+            yield from self.config.dependencies
+        for record in self._children:
+            yield from record.iter_source_filenames()
+
+    # -----------------------
+    #   Properties & Helper
+    # -----------------------
+
+    @property
+    def children(self) -> Dict['Record', List[Any]]:
+        ''' Returns dict with page record key and (optional) extra value. '''
+        return self._children
+
+    @property
+    def first_child(self) -> Optional['Record']:
+        ''' Returns first referencing page record. '''
+        if self._children:
+            return iter(self._children).__next__()
+        return None
+
+    @property
+    def first_extra(self) -> Optional[Any]:
+        ''' Returns first additional / extra info object of first page. '''
+        if not self._children:
+            return None
+        val = iter(self._children.values()).__next__()
+        return val[0] if val else None
+
+    def __getitem__(self, key: str) -> Any:
+        # Used for virtual path resolver
+        if key in ('_path', '_alt'):
+            return getattr(self, key[1:])
+        return self.__missing__(key)  # type: ignore[attr-defined]
+
+    def __lt__(self, other: 'GroupBySource') -> bool:
+        # Used for |sort filter ("group" is the provided original string)
+        return self.group.lower() < other.group.lower()
+
+    def __eq__(self, other: object) -> bool:
+        # Used for |unique filter
+        if self is other:
+            return True
+        return isinstance(other, GroupBySource) and \
+            self.path == other.path and self.slug == other.slug
+
+    def __hash__(self) -> int:
+        # Used for hashing in set and dict
+        return hash((self.path, self.slug))
+
+    def __repr__(self) -> str:
+        return '<GroupBySource path="{}" children={}>'.format(
+            self.path, len(self._children))
+
+
+# -----------------------------------
+#           BuildProgram
+# -----------------------------------
+
+class GroupByBuildProgram(BuildProgram):
+    ''' Generate Build-Artifacts and write files. '''
+
+    def produce_artifacts(self) -> None:
+        url = self.source.url_path
+        if url.endswith('/'):
+            url += 'index.html'
+        self.declare_artifact(url, sources=list(
+            self.source.iter_source_filenames()))
+
+    def build_artifact(self, artifact: 'Artifact') -> None:
+        get_ctx().record_virtual_dependency(self.source)
+        artifact.render_template_into(
+            self.source.config.template, this=self.source)

lektor_groupby/watcher.py

@@ -0,0 +1,111 @@
+from typing import TYPE_CHECKING, Dict, List, Tuple, Any, Union, NamedTuple
+from typing import Optional, Callable, Iterator, Generator
+from .backref import VGroups
+from .model import ModelReader
+from .vobj import GroupBySource
+if TYPE_CHECKING:
+    from lektor.db import Pad, Record
+    from .config import Config
+    from .model import FieldKeyPath
+
+
+class GroupByCallbackArgs(NamedTuple):
+    record: 'Record'
+    key: 'FieldKeyPath'
+    field: Any  # lektor model data-field value
+
+
+GroupingCallback = Callable[[GroupByCallbackArgs], Union[
+    Iterator[Union[str, Tuple[str, Any]]],
+    Generator[Union[str, Tuple[str, Any]], Optional[str], None],
+]]
+
+
+class Watcher:
+    '''
+    Callback is called with (Record, FieldKeyPath, field-value).
+    Callback may yield one or more (group, extra-info) tuples.
+    '''
+
+    def __init__(self, config: 'Config') -> None:
+        self.config = config
+        self._root = self.config.root
+
+    def grouping(self, flatten: bool = True) \
+            -> Callable[[GroupingCallback], None]:
+        '''
+        Decorator to subscribe to attrib-elements.
+        If flatten = False, dont explode FlowType.
+
+        (record, field-key, field) -> (group, extra-info)
+        '''
+        def _decorator(fn: GroupingCallback) -> None:
+            self.flatten = flatten
+            self.callback = fn
+        return _decorator
+
+    def initialize(self, pad: 'Pad') -> None:
+        ''' Reset internal state. You must initialize before each build! '''
+        assert callable(self.callback), 'No grouping callback provided.'
+        self._model_reader = ModelReader(pad.db, self.config.key, self.flatten)
+        self._root_record = pad.get(self._root)  # type: Record
+        self._state = {}  # type: Dict[str, GroupBySource]
+
+    def should_process(self, node: 'Record') -> bool:
+        ''' Check if record path is being watched. '''
+        return node['_path'].startswith(self._root)
+
+    def process(self, record: 'Record') -> None:
+        '''
+        Will iterate over all record fields and call the callback method.
+        Each record is guaranteed to be processed only once.
+        '''
+        for key, field in self._model_reader.read(record):
+            _gen = self.callback(GroupByCallbackArgs(record, key, field))
+            try:
+                obj = next(_gen)
+                while True:
+                    if not isinstance(obj, (str, tuple)):
+                        raise TypeError(f'Unsupported groupby yield: {obj}')
+                    slug = self._persist(record, key, obj)
+                    # return slugified group key and continue iteration
+                    if isinstance(_gen, Generator) and not _gen.gi_yieldfrom:
+                        obj = _gen.send(slug)
+                    else:
+                        obj = next(_gen)
+            except StopIteration:
+                del _gen
+
+    def _persist(
+        self, record: 'Record', key: 'FieldKeyPath', obj: Union[str, tuple]
+    ) -> str:
+        ''' Update internal state. Return slugified string. '''
+        if isinstance(obj, str):
+            group, extra = obj, key.fieldKey
+        else:
+            group, extra = obj
+
+        slug = self.config.slugify(group)
+        if slug not in self._state:
+            src = GroupBySource(self._root_record, slug)
+            self._state[slug] = src
+        else:
+            src = self._state[slug]
+
+        src.append_child(record, extra, group)
+        # reverse reference
+        VGroups.of(record).add(key, src)
+        return slug
+
+    def iter_sources(self, root: 'Record') -> Iterator[GroupBySource]:
+        ''' Prepare and yield GroupBySource elements. '''
+        for vobj in self._state.values():
+            yield vobj.finalize(self.config)
+        # cleanup. remove this code if you'd like to iter twice
+        del self._model_reader
+        del self._root_record
+        del self._state
+
+    def __repr__(self) -> str:
+        return '<GroupByWatcher key="{}" enabled={} callback={}>'.format(
+            self.config.key, self.config.enabled, self.callback)

setup.py

@@ -5,7 +5,7 @@ with open('README.md') as fp:
 
 setup(
     name='lektor-groupby',
-    py_modules=['lektor_groupby'],
+    packages=['lektor_groupby'],
     entry_points={
         'lektor.plugins': [
             'groupby = lektor_groupby:GroupByPlugin',
@@ -13,7 +13,7 @@ setup(
     },
     author='relikd',
     url='https://github.com/relikd/lektor-groupby-plugin',
-    version='0.9.1',
+    version='0.9.7',
     description='Cluster arbitrary records with field attribute keyword.',
     long_description=longdesc,
     long_description_content_type="text/markdown",
@@ -27,7 +27,6 @@ setup(
         'cluster',
     ],
     classifiers=[
-        'Development Status :: 5 - Production/Stable',
         'Environment :: Web Environment',
         'Environment :: Plugins',
         'Framework :: Lektor',