Compare commits
16 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
b05dd31ff0 | ||
|
|
16a26afdce | ||
|
|
c618ee458b | ||
|
|
55916a4519 | ||
|
|
a694149d04 | ||
|
|
831cfa4e9c | ||
|
|
298e0d4a62 | ||
|
|
2a6bdf05fd | ||
|
|
df4be7c60a | ||
|
|
637524a615 | ||
|
|
a6d9f715f9 | ||
|
|
d6df547682 | ||
|
|
ebc29459ec | ||
|
|
adb26e343e | ||
|
|
97b40b4886 | ||
|
|
479ff9b964 |
@@ -5,11 +5,10 @@ Can be used for tagging or similar tasks.
|
||||
The grouping algorithm is performed once.
|
||||
Contrary to, at least, cubic runtime if doing the same with Pad queries.
|
||||
|
||||
To install this plugin, modify your Lektor project file:
|
||||
Install this plugin or modify your Lektor project file:
|
||||
|
||||
```ini
|
||||
[packages]
|
||||
lektor-groupby = 0.9.1
|
||||
```sh
|
||||
lektor plugin add groupby
|
||||
```
|
||||
|
||||
Optionally, enable a basic config:
|
||||
|
||||
@@ -2,4 +2,4 @@
|
||||
name = GroupBy Examples
|
||||
|
||||
[packages]
|
||||
lektor-groupby = 0.9.1
|
||||
lektor-groupby = 0.9.3
|
||||
|
||||
7
examples/Makefile
Normal file
7
examples/Makefile
Normal file
@@ -0,0 +1,7 @@
|
||||
.PHONY: server clean plugins
|
||||
server:
|
||||
lektor server
|
||||
clean:
|
||||
lektor clean --yes -v
|
||||
plugins:
|
||||
lektor plugins flush-cache && lektor plugins list
|
||||
@@ -1,16 +1,19 @@
|
||||
# Usage
|
||||
|
||||
Overview:
|
||||
- the [quick config](#quick-config) example shows how you can use the plugin config to setup a quick and easy tagging system.
|
||||
- the [simple example](#simple-example) goes into detail how this plugin works.
|
||||
- the [advanced example](#advanced-example) touches on the potentials of the plugin.
|
||||
- [quick config example](#quick-config) shows how you can use the plugin config to setup a quick and easy tagging system.
|
||||
- [simple example](#simple-example) goes into detail how to use it in your own plugin.
|
||||
- [advanced example](#advanced-example) touches on the potentials of the plugin.
|
||||
- [Misc](#misc) shows other use-cases.
|
||||
|
||||
After reading this tutorial, have a look at other plugins that use `lektor-groupby`:
|
||||
- [lektor-inlinetags](https://github.com/relikd/lektor-inlinetags-plugin)
|
||||
|
||||
|
||||
## About
|
||||
|
||||
To use the groupby plugin you have to add an attribute to your model file.
|
||||
In our case you can refer to the `models/page.ini` model:
|
||||
In our case you can refer to the [`models/page.ini`](./models/page.ini) model:
|
||||
|
||||
```ini
|
||||
[fields.tags]
|
||||
@@ -35,19 +38,26 @@ The attribute name is later used for grouping.
|
||||
## Quick config
|
||||
|
||||
Relevant files:
|
||||
```
|
||||
configs/groupby.ini
|
||||
templates/example-config.html
|
||||
```
|
||||
|
||||
- [`configs/groupby.ini`](./configs/groupby.ini)
|
||||
- [`templates/example-config.html`](./templates/example-config.html)
|
||||
|
||||
|
||||
The easiest way to add tags to your site is by defining the `groupby.ini` config file.
|
||||
|
||||
```ini
|
||||
[testA]
|
||||
root = /
|
||||
slug = config/{group}.html
|
||||
slug = config/{key}.html
|
||||
template = example-config.html
|
||||
split = ' '
|
||||
enabled = True
|
||||
|
||||
[testA.fields]
|
||||
title = "Tagged: " ~ this.group
|
||||
|
||||
[testA.key_map]
|
||||
Blog = News
|
||||
```
|
||||
|
||||
The configuration parameter are:
|
||||
@@ -57,9 +67,11 @@ The configuration parameter are:
|
||||
All results will be placed under this directory, e.g., `/tags/tagname/`.
|
||||
If you use `root = /blog`, the results path will be `/blog/tags/tagname/`.
|
||||
The groupby plugin will traverse all sub-pages wich contain the attribute `testA`.
|
||||
3. The `slug` parameter (`config/{group}.html`) is where the results are placed.
|
||||
3. The `slug` parameter (`config/{key}.html`) is where the results are placed.
|
||||
In our case, the path resolves to `config/tagname.html`.
|
||||
The default value is `{attrib}/{group}/index.html` which would resolve to `testA/tagname/index.html`.
|
||||
The default value is `{attrib}/{key}/index.html` which would resolve to `testA/tagname/index.html`.
|
||||
If this field contains `{key}`, it just replaces the value with the group-key.
|
||||
In all other cases the field value is evaluated in a jinja context.
|
||||
4. The `template`parameter (`example-config.html`) is used to render the results page.
|
||||
If no explicit template is set, the default template `groupby-testA.html` will be used.
|
||||
Where `testA` is replaced with whatever attribute you chose.
|
||||
@@ -68,18 +80,54 @@ The configuration parameter are:
|
||||
The split is only relevant for fields of type `string` or `text`.
|
||||
These single-line fields are then expanded to lists as well.
|
||||
If you do not provide the `split` option, the whole field value will be used as tagname.
|
||||
6. The `enabled` parameter allows you to quickly disable the grouping.
|
||||
|
||||
|
||||
You can have multiple listeners, e.g., one for `/blog/` and another for `/projects/`.
|
||||
Just create as many custom attributes as you like, each having its own section.
|
||||
|
||||
In your template file you have access to the children (pages) and their tags.
|
||||
The emitted `extras` for the child is a list of original tagnames.
|
||||
There are two additional config mappings, `.fields` and `.key_map`.
|
||||
Key-value pairs in `.fields` will be added as attributes to your grouping.
|
||||
You can access them in your template (e.g., `{{this.title}}`).
|
||||
All of the `.fields` values are evaluted in a jinja context, so be cautious when using plain strings.
|
||||
|
||||
The built-in field attributes are:
|
||||
|
||||
- `group`: returned group name, e.g., "A Title?"
|
||||
- `key`: slugified group value, e.g., "a-title"
|
||||
- `slug`: url path after root node, e.g. "config/a-title.html" (can be `None`)
|
||||
- `record`: parent node, e.g., `Page(path="/")`
|
||||
- `children`: dictionary of `{record: extras}` pairs
|
||||
- `first_child`: first page
|
||||
- `first_extra`: first extra
|
||||
- `config`: configuration object (see below)
|
||||
|
||||
Without any changes, the `key` value will just be `slugify(group)`.
|
||||
However, the other mapping `.key_map` will replace `group` with whatever replacement value is provided in the `.key_map` and then slugified.
|
||||
You could, for example, add a `C# = c-sharp` mapping, which would otherwise just be slugified to `c`.
|
||||
This is equivalent to `slugify(key_map.get(group))`.
|
||||
|
||||
The `config` attribute contains the values that created the group:
|
||||
|
||||
- `key`: attribute key, e.g., `TestA`
|
||||
- `root`: as provided by init, e.g., `/`
|
||||
- `slug`: the raw value, e.g., `config/{key}.html`
|
||||
- `template`: as provided by init, e.g., `example-config.html`
|
||||
- `enabled`: boolean
|
||||
- `dependencies`: path to config file (if initialized from config)
|
||||
- `fields`: raw values from `TestA.fields`
|
||||
- `key_map`: raw values from `TestA.key_map`
|
||||
|
||||
In your template file you have access to the attributes, config, and children (pages):
|
||||
|
||||
```jinja2
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<li>Page: {{ child.path }}, Tags: {{ extras }}</li>
|
||||
<h2>{{ this.title }}</h2>
|
||||
<p>Key: {{ this.key }}, Attribute: {{ this.config.key }}</p>
|
||||
<ul>
|
||||
{%- for child in this.children %}
|
||||
<li>Page: {{ child.path }}</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
```
|
||||
|
||||
|
||||
@@ -87,66 +135,70 @@ The emitted `extras` for the child is a list of original tagnames.
|
||||
## Simple example
|
||||
|
||||
Relevant files:
|
||||
```
|
||||
packages/simple-example/lektor_simple.py
|
||||
templates/example-simple.html
|
||||
```
|
||||
|
||||
- [`packages/simple-example/lektor_simple.py`](./packages/simple-example/lektor_simple.py)
|
||||
- [`templates/example-simple.html`](./templates/example-simple.html)
|
||||
|
||||
|
||||
```python
|
||||
def on_groupby_after_build_all(self, groupby, builder, **extra):
|
||||
@groupby.watch('/blog', 'testB', slug='simple/{group}/index.html',
|
||||
template='example-simple.html', flatten=True)
|
||||
def convert_simple_example(args):
|
||||
value = args.field # list, since model is 'strings' type
|
||||
for tag in value:
|
||||
yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
|
||||
def on_groupby_before_build_all(self, groupby, builder, **extra):
|
||||
watcher = groupby.add_watcher('testB', {
|
||||
'root': '/blog',
|
||||
'slug': 'simple/{key}/index.html',
|
||||
'template': 'example-simple.html',
|
||||
})
|
||||
watcher.config.set_key_map({'Foo': 'bar'})
|
||||
watcher.config.set_fields({'date': datetime.now()})
|
||||
|
||||
# page = args.record # extract additional info from source
|
||||
# fieldKey, flowIndex, flowKey = args.key # or get field index
|
||||
# if flowIndex is None:
|
||||
# obj = page[fieldKey]
|
||||
# else:
|
||||
# obj = page[fieldKey].blocks[flowIndex].get(flowKey)
|
||||
@watcher.grouping(flatten=True)
|
||||
def convert_simple_example(args):
|
||||
# Yield groups
|
||||
value = args.field # type: list # since model is 'strings' type
|
||||
for tag in value:
|
||||
yield tag, {'tags_in_page': value}
|
||||
```
|
||||
|
||||
This example is roughly equivalent to the config file example.
|
||||
The parameters of the `@groupby.watch` function (`root`, `attribute`, `slug`, `template`) correspond to the same config parameters described above.
|
||||
There is a new `flatten` parameter:
|
||||
This example is roughly equivalent to the config example above – the parameters of the `groupby.add_watcher` function correspond to the same config parameters.
|
||||
Additionally, you can set other types in `set_fields` (all strings are evaluated in jinja context!).
|
||||
|
||||
- Flatten determines how Flow elements are processed.
|
||||
`@watcher.grouping` sets the callback to generate group keys.
|
||||
It has one optional flatten parameter:
|
||||
|
||||
- `flatten` determines how Flow elements are processed.
|
||||
If `False`, the callback function is called once per Flow element.
|
||||
If `True` (default), the callback is called for all Flow-Blocks of the Flow individually.
|
||||
The attribute `testB` can be attached to either the Flow or a Flow-Block regardless.
|
||||
|
||||
The `args` parameter of the `convert_simple_example()` function is a named tuple, it has three attributes:
|
||||
|
||||
1. The `record` points to the `Page` source which contains the tag.
|
||||
1. The `record` points to the `Page` record that contains the tag.
|
||||
2. The `key` tuple `(field-key, flow-index, flow-key)` tells which field is processed.
|
||||
For Flow types, `flow-index` and `flow-key` are set, otherwise they are `None`.
|
||||
3. The `field` value is the content of the processed field.
|
||||
The field value is reoughly equivalent to the following:
|
||||
The field value is equivalent to the following:
|
||||
|
||||
```python
|
||||
args.page[fieldKey].blocks[flowIndex].get(flowKey)
|
||||
k = args.key
|
||||
field = args.record[k.fieldKey].blocks[k.flowIndex].get(k.flowKey)
|
||||
```
|
||||
|
||||
The callback body **can** produce groupings but does not have to.
|
||||
If you choose to produce an entry, you have to `yield` a tuple pair of `(groupkey, extra-info)`.
|
||||
`groupkey` is used to combine & cluster pages and must be URL-safe.
|
||||
The `extra-info` is passed through to your template file.
|
||||
You can yield more than one entry per source or filter / ignore pages if you don't yield anything.
|
||||
If you choose to produce an entry, you have to `yield` a string or tuple pair `(group, extra-info)`.
|
||||
`group` is slugified (see above) and then used to combine & cluster pages.
|
||||
The `extra-info` (optional) is passed through to your template file.
|
||||
You can yield more than one entry per source.
|
||||
Or ignore pages if you don't yield anything.
|
||||
|
||||
The template file can access and display the `extra-info`:
|
||||
|
||||
```jinja2
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<b>Page: {{ child.title }}<b>
|
||||
<p>Custom field date: {{this.date}}</p>
|
||||
<ul>
|
||||
{%- for extra in extras %}
|
||||
<li>Name: {{ extra.val }}, Tag count: {{ extra.tags_in_page }}</li>
|
||||
{%- for child, extras in this.children.items() -%}
|
||||
{%- set etxra = (extras|first).tags_in_page %}
|
||||
<li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
{%- endfor %}
|
||||
```
|
||||
|
||||
|
||||
@@ -154,11 +206,11 @@ The template file can access and display the `extra-info`:
|
||||
## Advanced example
|
||||
|
||||
Relevant files:
|
||||
```
|
||||
configs/advanced.ini
|
||||
packages/advanced-example/lektor_advanced.py
|
||||
templates/example-advanced.html
|
||||
```
|
||||
|
||||
- [`configs/advanced.ini`](./configs/advanced.ini)
|
||||
- [`packages/advanced-example/lektor_advanced.py`](./packages/advanced-example/lektor_advanced.py)
|
||||
- [`templates/example-advanced.html`](./templates/example-advanced.html)
|
||||
|
||||
|
||||
The following example is similar to the previous one.
|
||||
Except that it loads a config file and replaces in-text occurrences of `{{Tagname}}` with `<a href="/tag/">Tagname</a>`.
|
||||
@@ -166,44 +218,72 @@ Except that it loads a config file and replaces in-text occurrences of `{{Tagnam
|
||||
```python
|
||||
def on_groupby_before_build_all(self, groupby, builder, **extra):
|
||||
# load config
|
||||
regex = re.compile(self.get_config().get('match'))
|
||||
# since we load and use a config file, we need to track the dependency
|
||||
@groupby.depends_on(self.config_filename)
|
||||
@groupby.watch('/', 'testC', slug='advanced/{group}/',
|
||||
template='example-advanced.html')
|
||||
config = self.get_config()
|
||||
regex = config.get('testC.pattern.match')
|
||||
try:
|
||||
regex = re.compile(regex)
|
||||
except Exception as e:
|
||||
print('inlinetags.regex not valid: ' + str(e))
|
||||
return
|
||||
|
||||
# load config directly (which also tracks dependency)
|
||||
watcher = groupby.add_watcher('testC', config)
|
||||
|
||||
@watcher.grouping()
|
||||
def convert_replace_example(args):
|
||||
# args.field assumed to be Markdown
|
||||
obj = args.field.source
|
||||
slugify_map = {} # type Dict[str, str]
|
||||
for match in regex.finditer(obj):
|
||||
tag = match.group(1)
|
||||
yield slugify(tag), tag
|
||||
key = yield tag
|
||||
print('[advanced] slugify:', tag, '->', key)
|
||||
slugify_map[tag] = key
|
||||
|
||||
def _fn(match: re.Match) -> str:
|
||||
tag = match.group(1)
|
||||
return f'<a href="/advanced/{slugify(tag)}/">{tag}</a>'
|
||||
return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
|
||||
args.field.source = regex.sub(_fn, obj)
|
||||
```
|
||||
|
||||
One **important** thing to notice is, we use `on_groupby_before_build_all` to register our callback function.
|
||||
This is required because we would like to modify the source **before** it is written to disk.
|
||||
If you look back to the [simple example](#simple-example), we used `on_groupby_after_build_all` because we did not care when it is executed.
|
||||
Generally, it makes little difference which one you use (`on-after` is likely less busy).
|
||||
Just know that you can process the source before or after it is build.
|
||||
Notice, `add_watcher` accepts a config file as parameter which keeps also track of dependencies and rebuilds pages when you edit the config file.
|
||||
Further, the `yield` call returns the slugified group-key.
|
||||
First, you do not need to slugify it yourself and second, potential replacements from `key_map` are already handled.
|
||||
|
||||
For Markdown fields, we can modify the `source` attribute directly.
|
||||
All other field typed need to be accessed via `args.record` key indirection.
|
||||
All other field types need to be accessed via `args.record` key indirection (see [simple example](#simple-example)).
|
||||
|
||||
```ini
|
||||
[testC]
|
||||
root = /
|
||||
slug = "advanced/{}/".format(this.key)
|
||||
template = example-advanced.html
|
||||
|
||||
[testC.pattern]
|
||||
match = {{([^}]{1,32})}}
|
||||
```
|
||||
|
||||
Lastly, the config file contains a regular expression which matches `{{` + any string less than 32 characters + `}}`.
|
||||
Notice, the parenthesis (`()`) will match the inner part but the replace function (`re.sub`) will remove the `{{}}` too.
|
||||
The config file takes the same parameters as the [config example](#quick-config).
|
||||
As you can see, `slug` is evaluated in jinja context.
|
||||
|
||||
If the user changes the regex pattern in the config file, we need to rebuild all tags.
|
||||
For this purpose we need to track changes to the config file.
|
||||
This is done by calling:
|
||||
We introduced a new config option `testC.pattern.match`.
|
||||
This regular expression matches `{{` + any string less than 32 characters + `}}`.
|
||||
Notice, the parenthesis (`()`) will match only the inner part but the replace function (`re.sub`) will remove the `{{}}`.
|
||||
|
||||
```python
|
||||
@groupby.depends_on(file1, file2, ...)
|
||||
|
||||
|
||||
## Misc
|
||||
|
||||
It was shortly mentioned above that slugs can be `None` (only if manually set to `slug = None`).
|
||||
This is useful if you do not want to create subpages but rather an index page containing all groups.
|
||||
This can be done in combination with the next use-case:
|
||||
|
||||
```jinja2
|
||||
{%- for x in this|vgroups('TestA', 'TestB', recursive=True)|unique|sort %}
|
||||
<a href="{{ x|url }}">({{ x.group }})</a>
|
||||
{%- endfor %}
|
||||
```
|
||||
|
||||
You can query the groups of any parent node (including those without slug).
|
||||
The keys (`'TestA', 'TestB'`) can be omitted which will return all groups of all attributes (you can still filter them with `x.config.key == 'TestC'`).
|
||||
Refer to [`templates/page.html`](./templates/page.html) for usage.
|
||||
|
||||
@@ -1 +1,15 @@
|
||||
match = {{([^}]{1,32})}}
|
||||
[testC]
|
||||
root = /
|
||||
slug = "advanced/{}/".format(this.key)
|
||||
template = example-advanced.html
|
||||
|
||||
[testC.pattern]
|
||||
match = {{([^}]{1,32})}}
|
||||
|
||||
[testC.fields]
|
||||
desc = "Tag: " ~ this.group ~ ", Key: " ~ this.key
|
||||
|
||||
[testC.key_map]
|
||||
Blog = case-sensitive
|
||||
Two = three
|
||||
three = no-nested-replace
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
[testA]
|
||||
enabled = True
|
||||
root = /
|
||||
slug = config/{group}.html
|
||||
slug = config/{key}.html
|
||||
template = example-config.html
|
||||
split = ' '
|
||||
|
||||
[testA.fields]
|
||||
title = "Tagged: " ~ this.group
|
||||
|
||||
[testA.key_map]
|
||||
Blog = News
|
||||
|
||||
@@ -1,31 +1,36 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from lektor.pluginsystem import Plugin
|
||||
from lektor.utils import slugify
|
||||
from typing import Generator
|
||||
import re
|
||||
from lektor_groupby import GroupBy, GroupByCallbackArgs
|
||||
|
||||
|
||||
class AdvancedGroupByPlugin(Plugin):
|
||||
def on_groupby_before_build_all(self, groupby, builder, **extra):
|
||||
def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
|
||||
# load config
|
||||
regex = self.get_config().get('match')
|
||||
config = self.get_config()
|
||||
regex = config.get('testC.pattern.match')
|
||||
try:
|
||||
regex = re.compile(regex)
|
||||
except Exception as e:
|
||||
print('inlinetags.regex not valid: ' + str(e))
|
||||
return
|
||||
|
||||
# since we load and use a config file, we need to track the dependency
|
||||
@groupby.depends_on(self.config_filename)
|
||||
@groupby.watch('/', 'testC', slug='advanced/{group}/',
|
||||
template='example-advanced.html')
|
||||
def convert_replace_example(args):
|
||||
# load config directly (which also tracks dependency)
|
||||
watcher = groupby.add_watcher('testC', config)
|
||||
|
||||
@watcher.grouping()
|
||||
def _replace(args: GroupByCallbackArgs) -> Generator[str, str, None]:
|
||||
# args.field assumed to be Markdown
|
||||
obj = args.field.source
|
||||
slugify_map = {} # type Dict[str, str]
|
||||
for match in regex.finditer(obj):
|
||||
tag = match.group(1)
|
||||
yield slugify(tag), tag
|
||||
key = yield tag
|
||||
print('[advanced] slugify:', tag, '->', key)
|
||||
slugify_map[tag] = key
|
||||
|
||||
def _fn(match: re.Match) -> str:
|
||||
tag = match.group(1)
|
||||
return f'<a href="/advanced/{slugify(tag)}/">{tag}</a>'
|
||||
return f'<a href="/advanced/{slugify_map[tag]}/">{tag}</a>'
|
||||
args.field.source = regex.sub(_fn, obj)
|
||||
|
||||
@@ -1,17 +1,26 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from lektor.pluginsystem import Plugin
|
||||
from lektor.utils import slugify
|
||||
from typing import Iterator, Tuple
|
||||
from datetime import datetime
|
||||
from lektor_groupby import GroupBy, GroupByCallbackArgs
|
||||
|
||||
|
||||
class SimpleGroupByPlugin(Plugin):
|
||||
def on_groupby_after_build_all(self, groupby, builder, **extra):
|
||||
@groupby.watch('/blog', 'testB', slug='simple/{group}/index.html',
|
||||
template='example-simple.html', flatten=True)
|
||||
def convert_simple_example(args):
|
||||
def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
|
||||
watcher = groupby.add_watcher('testB', {
|
||||
'root': '/blog',
|
||||
'slug': 'simple/{key}/index.html',
|
||||
'template': 'example-simple.html',
|
||||
})
|
||||
watcher.config.set_key_map({'Foo': 'bar'})
|
||||
watcher.config.set_fields({'date': datetime.now()})
|
||||
|
||||
@watcher.grouping(flatten=True)
|
||||
def fn_simple(args: GroupByCallbackArgs) -> Iterator[Tuple[str, dict]]:
|
||||
# Yield groups
|
||||
value = args.field # list type since model is 'strings' type
|
||||
value = args.field # type: list # since model is 'strings' type
|
||||
for tag in value:
|
||||
yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
|
||||
yield tag, {'tags_in_page': value}
|
||||
# Everything below is just for documentation purposes
|
||||
page = args.record # extract additional info from source
|
||||
fieldKey, flowIndex, flowKey = args.key # or get field index
|
||||
@@ -19,6 +28,6 @@ class SimpleGroupByPlugin(Plugin):
|
||||
obj = page[fieldKey]
|
||||
else:
|
||||
obj = page[fieldKey].blocks[flowIndex].get(flowKey)
|
||||
print('page:', page)
|
||||
print(' obj:', obj)
|
||||
print()
|
||||
print('[simple] page:', page)
|
||||
print('[simple] obj:', obj)
|
||||
print('[simple] ')
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
<h2>Path: {{ this | url(absolute=True) }}</h2>
|
||||
<div>This is: {{this}}</div>
|
||||
<div>Children: {{this.children}}</div>
|
||||
<p>This is: {{this}}</p>
|
||||
<p>Custom field, desc: "{{this.desc}}"</p>
|
||||
<p>Children: {{this.children}}</p>
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
<h2>Path: {{ this | url(absolute=True) }}</h2>
|
||||
<div>This is: {{this}}</div>
|
||||
<p>This is: {{this}}</p>
|
||||
<p>Group: "{{this.group}}", Key: "{{this.key}}"</p>
|
||||
<p>Custom field title: {{this.title}}</p>
|
||||
<ul>
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<li>Page: {{ child.path }}, Tags: {{ extras }}</li>
|
||||
{%- endfor %}
|
||||
{%- for child in this.children %}
|
||||
<li>Child: <a href="{{child|url}}">{{child.title}}</a> ({{child.path}})</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
@@ -1,12 +1,9 @@
|
||||
<h2>Path: {{ this | url(absolute=True) }}</h2>
|
||||
<div>This is: {{this}}</div>
|
||||
<p>This is: {{this}}</p>
|
||||
<p>Custom field date: {{this.date}}</p>
|
||||
<ul>
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<li>Page: {{ child.path }}</li>
|
||||
<ul>
|
||||
{%- for extra in extras %}
|
||||
Name: {{ extra.val }}, Tag count: {{ extra.tags_in_page }}
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
{%- endfor %}
|
||||
{%- for child, extras in this.children.items() -%}
|
||||
{%- set etxra = (extras|first).tags_in_page %}
|
||||
<li>{{etxra|length}} tags on page "{{child.path}}": {{etxra}}</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
@@ -18,20 +18,12 @@ main { margin: 3em; }
|
||||
{% block body %}{{ this.body }}{% endblock %}
|
||||
</main>
|
||||
<footer>
|
||||
<div>Simple Tags:
|
||||
{% for tag in ['blog','directory','blog-post','initial','samegroup'] %}
|
||||
<a href="/blog/simple/{{tag}}/">({{tag}})</a>
|
||||
{% endfor %}
|
||||
</div>
|
||||
<div>Config Tags:
|
||||
{% for tag in ['root','blog','directory','blog-post','initial','samegroup'] %}
|
||||
<a href="/config/{{tag}}.html">({{tag}})</a>
|
||||
{% endfor %}
|
||||
</div>
|
||||
<div>Advanced Tags:
|
||||
{% for tag in ['tag','two','blog'] %}
|
||||
<a href="/advanced/{{tag}}/">({{tag}})</a>
|
||||
{% endfor %}
|
||||
{%- for k, v in [('testA','Config'),('testB','Simple'),('testC','Advanced')] %}
|
||||
<div>{{v}} Tags:
|
||||
{%- for x in this|vgroups(k, recursive=True)|unique|sort %}
|
||||
<a href="{{ x|url }}">({{x.key}})</a>
|
||||
{%- endfor %}
|
||||
</div>
|
||||
{%- endfor %}
|
||||
</footer>
|
||||
</body>
|
||||
|
||||
@@ -1,498 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from lektor.db import Database, Record # typing
|
||||
from lektor.build_programs import BuildProgram
|
||||
from lektor.builder import Artifact, Builder # typing
|
||||
from lektor.pluginsystem import Plugin
|
||||
from lektor.reporter import reporter
|
||||
from lektor.sourceobj import SourceObject, VirtualSourceObject
|
||||
from lektor.types.flow import Flow, FlowType
|
||||
from lektor.utils import bool_from_string, build_url, prune_file_and_folder
|
||||
# for quick config
|
||||
from lektor.utils import slugify
|
||||
|
||||
from typing import Tuple, Dict, Set, List, NamedTuple
|
||||
from typing import NewType, Optional, Iterator, Callable, Iterable
|
||||
|
||||
VPATH = '@groupby' # potentially unsafe. All matching entries are pruned.
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Typing
|
||||
# -----------------------------------
|
||||
AttributeKey = NewType('AttributeKey', str) # attribute of lektor model
|
||||
GroupKey = NewType('GroupKey', str) # key of group-by
|
||||
|
||||
|
||||
class ResolverConf(NamedTuple):
|
||||
attrib: AttributeKey
|
||||
group: GroupKey
|
||||
slug: str
|
||||
|
||||
|
||||
class FieldKeyPath(NamedTuple):
|
||||
fieldKey: str
|
||||
flowIndex: Optional[int] = None
|
||||
flowKey: Optional[str] = None
|
||||
|
||||
|
||||
class GroupByCallbackArgs(NamedTuple):
|
||||
record: Record
|
||||
key: FieldKeyPath
|
||||
field: object # lektor model data-field value
|
||||
|
||||
|
||||
GroupingCallback = Callable[[GroupByCallbackArgs],
|
||||
Iterator[Tuple[GroupKey, object]]]
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# VirtualSource & BuildProgram
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupBySource(VirtualSourceObject):
|
||||
'''
|
||||
Holds information for a single group/cluster.
|
||||
This object is accessible in your template file.
|
||||
Attributes: record, attrib, group, slug, template, children
|
||||
|
||||
:DEFAULTS:
|
||||
slug: "{attrib}/{group}/index.html"
|
||||
template: "groupby-attribute.html"
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
record: Record,
|
||||
attrib: AttributeKey,
|
||||
group: GroupKey, *,
|
||||
slug: Optional[str] = None, # default: "{attrib}/{group}/index.html"
|
||||
template: Optional[str] = None # default: "groupby-attrib.html"
|
||||
) -> None:
|
||||
super().__init__(record)
|
||||
self.attrib = attrib
|
||||
self.group = group
|
||||
self.template = template or 'groupby-{}.html'.format(self.attrib)
|
||||
# custom user path
|
||||
slug = slug or '{attrib}/{group}/index.html'
|
||||
slug = slug.replace('{attrib}', self.attrib)
|
||||
slug = slug.replace('{group}', self.group)
|
||||
if slug.endswith('/index.html'):
|
||||
slug = slug[:-10]
|
||||
self.slug = slug
|
||||
# user adjustable after init
|
||||
self.children = {} # type: Dict[Record, List[object]]
|
||||
self.dependencies = set() # type: Set[str]
|
||||
|
||||
@property
|
||||
def path(self) -> str:
|
||||
# Used in VirtualSourceInfo, used to prune VirtualObjects
|
||||
return f'{self.record.path}{VPATH}/{self.attrib}/{self.group}'
|
||||
|
||||
@property
|
||||
def url_path(self) -> str:
|
||||
# Actual path to resource as seen by the browser
|
||||
return build_url([self.record.path, self.slug])
|
||||
|
||||
def iter_source_filenames(self) -> Iterator[str]:
|
||||
''' Enumerate all dependencies '''
|
||||
if self.dependencies:
|
||||
yield from self.dependencies
|
||||
for record in self.children:
|
||||
yield from record.iter_source_filenames()
|
||||
|
||||
def __str__(self) -> str:
|
||||
txt = '<GroupBySource'
|
||||
for x in ['attrib', 'group', 'slug', 'template']:
|
||||
txt += ' {}="{}"'.format(x, getattr(self, x))
|
||||
return txt + ' children={}>'.format(len(self.children))
|
||||
|
||||
|
||||
class GroupByBuildProgram(BuildProgram):
|
||||
''' Generate Build-Artifacts and write files. '''
|
||||
|
||||
def produce_artifacts(self) -> None:
|
||||
url = self.source.url_path
|
||||
if url.endswith('/'):
|
||||
url += 'index.html'
|
||||
self.declare_artifact(url, sources=list(
|
||||
self.source.iter_source_filenames()))
|
||||
GroupByPruner.track(url)
|
||||
|
||||
def build_artifact(self, artifact: Artifact) -> None:
|
||||
self.source.pad.db.track_record_dependency(self.source)
|
||||
artifact.render_template_into(self.source.template, this=self.source)
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Helper
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupByPruner:
|
||||
'''
|
||||
Static collector for build-artifact urls.
|
||||
All non-tracked VPATH-urls will be pruned after build.
|
||||
'''
|
||||
_cache: Set[str] = set()
|
||||
# Note: this var is static or otherwise two instances of
|
||||
# GroupByCreator would prune each others artifacts.
|
||||
|
||||
@classmethod
|
||||
def track(cls, url: str) -> None:
|
||||
''' Add url to build cache to prevent pruning. '''
|
||||
cls._cache.add(url.lstrip('/'))
|
||||
|
||||
@classmethod
|
||||
def prune(cls, builder: Builder) -> None:
|
||||
''' Remove previously generated, unreferenced Artifacts. '''
|
||||
dest_path = builder.destination_path
|
||||
con = builder.connect_to_database()
|
||||
try:
|
||||
with builder.new_build_state() as build_state:
|
||||
for url, file in build_state.iter_artifacts():
|
||||
if url.lstrip('/') in cls._cache:
|
||||
continue # generated in this build-run
|
||||
infos = build_state.get_artifact_dependency_infos(url, [])
|
||||
for v_path, _ in infos:
|
||||
if VPATH not in v_path:
|
||||
continue # we only care about groupby Virtuals
|
||||
reporter.report_pruned_artifact(url)
|
||||
prune_file_and_folder(file.filename, dest_path)
|
||||
build_state.remove_artifact(url)
|
||||
break # there is only one VPATH-entry per source
|
||||
finally:
|
||||
con.close()
|
||||
cls._cache.clear()
|
||||
|
||||
|
||||
class GroupByModelReader:
|
||||
''' Find models and flow-models which contain attrib '''
|
||||
|
||||
def __init__(self, db: Database, attrib: AttributeKey) -> None:
|
||||
self._flows = {} # type: Dict[str, Set[str]]
|
||||
self._models = {} # type: Dict[str, Dict[str, str]]
|
||||
# find flow blocks with attrib
|
||||
for key, flow in db.flowblocks.items():
|
||||
tmp1 = set(f.name for f in flow.fields
|
||||
if bool_from_string(f.options.get(attrib, False)))
|
||||
if tmp1:
|
||||
self._flows[key] = tmp1
|
||||
# find models with attrib or flow-blocks containing attrib
|
||||
for key, model in db.datamodels.items():
|
||||
tmp2 = {} # Dict[str, str]
|
||||
for field in model.fields:
|
||||
if bool_from_string(field.options.get(attrib, False)):
|
||||
tmp2[field.name] = '*' # include all children
|
||||
elif isinstance(field.type, FlowType) and self._flows:
|
||||
# only processed if at least one flow has attrib
|
||||
fbs = field.type.flow_blocks
|
||||
# if fbs == None, all flow-blocks are allowed
|
||||
if fbs is None or any(x in self._flows for x in fbs):
|
||||
tmp2[field.name] = '?' # only some flow blocks
|
||||
if tmp2:
|
||||
self._models[key] = tmp2
|
||||
|
||||
def read(
|
||||
self,
|
||||
record: Record,
|
||||
flatten: bool = False
|
||||
) -> Iterator[Tuple[FieldKeyPath, object]]:
|
||||
'''
|
||||
Enumerate all fields of a Record with attrib = True.
|
||||
Flows are either returned directly (flatten=False) or
|
||||
expanded so that each flow-block is yielded (flatten=True)
|
||||
'''
|
||||
assert isinstance(record, Record)
|
||||
for r_key, subs in self._models.get(record.datamodel.id, {}).items():
|
||||
if subs == '*': # either normal field or flow type (all blocks)
|
||||
field = record[r_key]
|
||||
if flatten and isinstance(field, Flow):
|
||||
for i, flow in enumerate(field.blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key, block in flow._data.items():
|
||||
if f_key.startswith('_'): # e.g., _flowblock
|
||||
continue
|
||||
yield FieldKeyPath(r_key, i, f_key), block
|
||||
else:
|
||||
yield FieldKeyPath(r_key), field
|
||||
else: # always flow type (only some blocks)
|
||||
for i, flow in enumerate(record[r_key].blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key in self._flows.get(flowtype, []):
|
||||
yield FieldKeyPath(r_key, i, f_key), flow[f_key]
|
||||
|
||||
|
||||
class GroupByState:
|
||||
''' Holds and updates a groupby build state. '''
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.state = {} # type: Dict[GroupKey, Dict[Record, List]]
|
||||
self._processed = set() # type: Set[Record]
|
||||
|
||||
def __contains__(self, record: Record) -> bool:
|
||||
''' Returns True if record was already processed. '''
|
||||
return record.path in self._processed
|
||||
|
||||
def items(self) -> Iterable[Tuple[GroupKey, Dict]]:
|
||||
''' Iterable with (GroupKey, {record: extras}) tuples. '''
|
||||
return self.state.items()
|
||||
|
||||
def add(self, record: Record, group: Dict[GroupKey, List]) -> None:
|
||||
''' Append groups if not processed already. '''
|
||||
if record.path not in self._processed:
|
||||
self._processed.add(record.path)
|
||||
for group_key, extras in group.items():
|
||||
if group_key in self.state:
|
||||
self.state[group_key][record] = extras
|
||||
else:
|
||||
self.state[group_key] = {record: extras}
|
||||
|
||||
|
||||
class GroupByWatcher:
|
||||
'''
|
||||
Callback is called with (Record, FieldKeyPath, field-value).
|
||||
Callback may yield one or more (group-key, extra-info) tuples.
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
root: str,
|
||||
attrib: AttributeKey,
|
||||
callback: GroupingCallback, *,
|
||||
slug: Optional[str] = None, # default: "{attrib}/{group}/index.html"
|
||||
template: Optional[str] = None # default: "groupby-attrib.html"
|
||||
) -> None:
|
||||
self.root = root
|
||||
self.attrib = attrib
|
||||
self.callback = callback
|
||||
self.slug = slug
|
||||
self.template = template
|
||||
# user editable attributes
|
||||
self.flatten = True # if False, dont explode FlowType
|
||||
self.dependencies = set() # type: Set[str]
|
||||
|
||||
def initialize(self, db: Database) -> None:
|
||||
''' Reset internal state. You must initialize before each build! '''
|
||||
self._state = GroupByState()
|
||||
self._model_reader = GroupByModelReader(db, self.attrib)
|
||||
|
||||
def should_process(self, node: SourceObject) -> bool:
|
||||
''' Check if record path is being watched. '''
|
||||
if isinstance(node, Record):
|
||||
p = node['_path'] # type: str
|
||||
return p.startswith(self.root) or p + '/' == self.root
|
||||
return False
|
||||
|
||||
def process(self, record: Record) -> None:
|
||||
'''
|
||||
Will iterate over all record fields and call the callback method.
|
||||
Each record is guaranteed to be processed only once.
|
||||
'''
|
||||
if record in self._state:
|
||||
return
|
||||
tmp = {}
|
||||
for key, field in self._model_reader.read(record, self.flatten):
|
||||
for ret in self.callback(GroupByCallbackArgs(record, key, field)):
|
||||
assert isinstance(ret, (tuple, list)), \
|
||||
'Must return tuple (group-key, extra-info)'
|
||||
group_key, extra = ret
|
||||
if group_key not in tmp:
|
||||
tmp[group_key] = [extra]
|
||||
else:
|
||||
tmp[group_key].append(extra)
|
||||
self._state.add(record, tmp)
|
||||
|
||||
def iter_sources(self, root: Record) -> Iterator[GroupBySource]:
|
||||
''' Prepare and yield GroupBySource elements. '''
|
||||
for group_key, children in self._state.items():
|
||||
src = GroupBySource(root, self.attrib, group_key,
|
||||
slug=self.slug, template=self.template)
|
||||
src.dependencies = self.dependencies
|
||||
src.children = children
|
||||
yield src
|
||||
|
||||
def __str__(self) -> str:
|
||||
txt = '<GroupByWatcher'
|
||||
for x in [
|
||||
'root', 'attrib', 'slug', 'template', 'flatten', 'dependencies'
|
||||
]:
|
||||
txt += ' {}="{}"'.format(x, getattr(self, x))
|
||||
return txt + '>'
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Main Component
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupByCreator:
|
||||
'''
|
||||
Process all children with matching conditions under specified page.
|
||||
Creates a grouping of pages with similar (self-defined) attributes.
|
||||
The grouping is performed only once per build.
|
||||
'''
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._watcher = [] # type: List[GroupByWatcher]
|
||||
self._results = {} # type: Dict[str, GroupBySource]
|
||||
self._resolve_map = {} # type: Dict[str, ResolverConf]
|
||||
|
||||
# ----------------
|
||||
# Add Observer
|
||||
# ----------------
|
||||
|
||||
def depends_on(self, *args: str) \
|
||||
-> Callable[[GroupByWatcher], GroupByWatcher]:
|
||||
''' Set GroupBySource dependency, e.g., a plugin config file. '''
|
||||
def _decorator(r: GroupByWatcher) -> GroupByWatcher:
|
||||
r.dependencies.update(list(args))
|
||||
return r
|
||||
return _decorator
|
||||
|
||||
def watch(
|
||||
self,
|
||||
root: str,
|
||||
attrib: AttributeKey, *,
|
||||
slug: Optional[str] = None, # default: "{attrib}/{group}/index.html"
|
||||
template: Optional[str] = None, # default: "groupby-attrib.html"
|
||||
flatten: bool = True, # if False, dont explode FlowType
|
||||
) -> Callable[[GroupingCallback], GroupByWatcher]:
|
||||
'''
|
||||
Decorator to subscribe to attrib-elements.
|
||||
(record, field-key, field) -> (group-key, extra-info)
|
||||
|
||||
:DEFAULTS:
|
||||
slug: "{attrib}/{group}/index.html"
|
||||
template: "groupby-attrib.html"
|
||||
'''
|
||||
root = root.rstrip('/') + '/'
|
||||
|
||||
def _decorator(fn: GroupingCallback) -> GroupByWatcher:
|
||||
w = GroupByWatcher(root, attrib, fn, slug=slug, template=template)
|
||||
w.flatten = flatten
|
||||
self._watcher.append(w)
|
||||
return w
|
||||
|
||||
return _decorator
|
||||
|
||||
# -----------
|
||||
# Builder
|
||||
# -----------
|
||||
|
||||
def clear_previous_results(self) -> None:
|
||||
''' Reset prvious results. Must be called before each build. '''
|
||||
self._watcher.clear()
|
||||
self._results.clear()
|
||||
self._resolve_map.clear()
|
||||
|
||||
def make_cluster(self, builder: Builder) -> None:
|
||||
''' Perform groupby, iterate over all children. '''
|
||||
if not self._watcher:
|
||||
return
|
||||
for w in self._watcher:
|
||||
w.initialize(builder.pad.db)
|
||||
|
||||
queue = builder.pad.get_all_roots() # type: List[SourceObject]
|
||||
while queue:
|
||||
record = queue.pop()
|
||||
self.queue_now(record)
|
||||
if hasattr(record, 'attachments'):
|
||||
queue.extend(record.attachments) # type: ignore[attr-defined]
|
||||
if hasattr(record, 'children'):
|
||||
queue.extend(record.children) # type: ignore[attr-defined]
|
||||
# build artifacts
|
||||
for w in self._watcher:
|
||||
root = builder.pad.get(w.root)
|
||||
for vobj in w.iter_sources(root):
|
||||
self._results[vobj.url_path] = vobj
|
||||
self._watcher.clear()
|
||||
|
||||
def queue_now(self, node: SourceObject) -> None:
|
||||
''' Process record immediatelly (No-Op if already processed). '''
|
||||
for w in self._watcher:
|
||||
if w.should_process(node): # ensures type Record
|
||||
w.process(node) # type: ignore[arg-type]
|
||||
|
||||
def build_all(self, builder: Builder) -> None:
|
||||
''' Create virtual objects and build sources. '''
|
||||
for url, x in sorted(self._results.items()):
|
||||
builder.build(x)
|
||||
self._resolve_map[url] = ResolverConf(x.attrib, x.group, x.slug)
|
||||
self._results.clear()
|
||||
|
||||
# -----------------
|
||||
# Path resolver
|
||||
# -----------------
|
||||
|
||||
def resolve_dev_server_path(
|
||||
self, node: SourceObject, pieces: List[str]
|
||||
) -> Optional[GroupBySource]:
|
||||
''' Dev server only: Resolves path/ -> path/index.html '''
|
||||
if not isinstance(node, Record):
|
||||
return None
|
||||
conf = self._resolve_map.get(build_url([node.url_path] + pieces))
|
||||
if not conf:
|
||||
return None
|
||||
return GroupBySource(node, conf.attrib, conf.group, slug=conf.slug)
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Plugin Entry
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupByPlugin(Plugin):
|
||||
name = 'GroupBy Plugin'
|
||||
description = 'Cluster arbitrary records with field attribute keyword.'
|
||||
|
||||
def on_setup_env(self, **extra: object) -> None:
|
||||
self.creator = GroupByCreator()
|
||||
self.env.add_build_program(GroupBySource, GroupByBuildProgram)
|
||||
|
||||
# resolve /tag/rss/ -> /tag/rss/index.html (local server only)
|
||||
@self.env.urlresolver
|
||||
def _(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
|
||||
return self.creator.resolve_dev_server_path(node, parts)
|
||||
|
||||
def _load_quick_config(self) -> None:
|
||||
''' Load config file quick listeners. '''
|
||||
config = self.get_config()
|
||||
for attrib in config.sections():
|
||||
sect = config.section_as_dict(attrib)
|
||||
root = sect.get('root', '/')
|
||||
slug = sect.get('slug')
|
||||
temp = sect.get('template')
|
||||
split = sect.get('split')
|
||||
|
||||
@self.creator.depends_on(self.config_filename)
|
||||
@self.creator.watch(root, attrib, slug=slug, template=temp)
|
||||
def _fn(args: GroupByCallbackArgs) \
|
||||
-> Iterator[Tuple[GroupKey, object]]:
|
||||
val = args.field
|
||||
if isinstance(val, str):
|
||||
val = val.split(split) if split else [val] # make list
|
||||
if isinstance(val, list):
|
||||
for tag in val:
|
||||
yield slugify(tag), tag
|
||||
|
||||
def on_before_build_all(self, builder: Builder, **extra: object) -> None:
|
||||
self.creator.clear_previous_results()
|
||||
# let other plugins register their @groupby.watch functions
|
||||
self.emit('before-build-all', groupby=self.creator, builder=builder)
|
||||
self.creator.make_cluster(builder)
|
||||
|
||||
def on_before_build(self, source: SourceObject, **extra: object) -> None:
|
||||
# before-build may be called before before-build-all (issue #1017)
|
||||
# make sure it is evaluated immediatelly
|
||||
self.creator.queue_now(source)
|
||||
|
||||
def on_after_build_all(self, builder: Builder, **extra: object) -> None:
|
||||
self.emit('after-build-all', groupby=self.creator, builder=builder)
|
||||
self._load_quick_config()
|
||||
self.creator.make_cluster(builder)
|
||||
self.creator.build_all(builder)
|
||||
|
||||
def on_after_prune(self, builder: Builder, **extra: object) -> None:
|
||||
# TODO: find a better way to prune unreferenced elements
|
||||
GroupByPruner.prune(builder)
|
||||
4
lektor_groupby/__init__.py
Normal file
4
lektor_groupby/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
from .config import Config # noqa: F401
|
||||
from .groupby import GroupBy # noqa: F401
|
||||
from .plugin import GroupByPlugin # noqa: F401
|
||||
from .watcher import GroupByCallbackArgs # noqa: F401
|
||||
87
lektor_groupby/config.py
Normal file
87
lektor_groupby/config.py
Normal file
@@ -0,0 +1,87 @@
|
||||
from inifile import IniFile
|
||||
from lektor.utils import slugify
|
||||
|
||||
from typing import Set, Dict, Optional, Union, Any
|
||||
|
||||
AnyConfig = Union['Config', IniFile, Dict]
|
||||
|
||||
|
||||
class Config:
|
||||
'''
|
||||
Holds information for GroupByWatcher and GroupBySource.
|
||||
This object is accessible in your template file ({{this.config}}).
|
||||
|
||||
Available attributes:
|
||||
key, root, slug, template, enabled, dependencies, fields, key_map
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
key: str, *,
|
||||
root: Optional[str] = None, # default: "/"
|
||||
slug: Optional[str] = None, # default: "{attr}/{group}/index.html"
|
||||
template: Optional[str] = None, # default: "groupby-{attr}.html"
|
||||
) -> None:
|
||||
self.key = key
|
||||
self.root = (root or '/').rstrip('/') or '/'
|
||||
self.slug = slug or (key + '/{key}/') # key = GroupBySource.key
|
||||
self.template = template or f'groupby-{self.key}.html'
|
||||
# editable after init
|
||||
self.enabled = True
|
||||
self.dependencies = set() # type: Set[str]
|
||||
self.fields = {} # type: Dict[str, Any]
|
||||
self.key_map = {} # type: Dict[str, str]
|
||||
|
||||
def slugify(self, k: str) -> str:
|
||||
''' key_map replace and slugify. '''
|
||||
return slugify(self.key_map.get(k, k)) # type: ignore[no-any-return]
|
||||
|
||||
def set_fields(self, fields: Optional[Dict[str, Any]]) -> None:
|
||||
'''
|
||||
The fields dict is a mapping of attrib = Expression values.
|
||||
Each dict key will be added to the GroupBySource virtual object.
|
||||
Each dict value is passed through jinja context first.
|
||||
'''
|
||||
self.fields = fields or {}
|
||||
|
||||
def set_key_map(self, key_map: Optional[Dict[str, str]]) -> None:
|
||||
''' This mapping replaces group keys before slugify. '''
|
||||
self.key_map = key_map or {}
|
||||
|
||||
def __repr__(self) -> str:
|
||||
txt = '<GroupByConfig'
|
||||
for x in ['key', 'root', 'slug', 'template', 'enabled']:
|
||||
txt += ' {}="{}"'.format(x, getattr(self, x))
|
||||
txt += f' fields="{", ".join(self.fields)}"'
|
||||
return txt + '>'
|
||||
|
||||
@staticmethod
|
||||
def from_dict(key: str, cfg: Dict[str, str]) -> 'Config':
|
||||
''' Set config fields manually. Allowed: key, root, slug, template. '''
|
||||
return Config(
|
||||
key=key,
|
||||
root=cfg.get('root'),
|
||||
slug=cfg.get('slug'),
|
||||
template=cfg.get('template'),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def from_ini(key: str, ini: IniFile) -> 'Config':
|
||||
''' Read and parse ini file. Also adds dependency tracking. '''
|
||||
cfg = ini.section_as_dict(key) # type: Dict[str, str]
|
||||
conf = Config.from_dict(key, cfg)
|
||||
conf.enabled = ini.get_bool(key + '.enabled', True)
|
||||
conf.dependencies.add(ini.filename)
|
||||
conf.set_fields(ini.section_as_dict(key + '.fields'))
|
||||
conf.set_key_map(ini.section_as_dict(key + '.key_map'))
|
||||
return conf
|
||||
|
||||
@staticmethod
|
||||
def from_any(key: str, config: AnyConfig) -> 'Config':
|
||||
assert isinstance(config, (Config, IniFile, Dict))
|
||||
if isinstance(config, Config):
|
||||
return config
|
||||
elif isinstance(config, IniFile):
|
||||
return Config.from_ini(key, config)
|
||||
elif isinstance(config, Dict):
|
||||
return Config.from_dict(key, config)
|
||||
116
lektor_groupby/groupby.py
Normal file
116
lektor_groupby/groupby.py
Normal file
@@ -0,0 +1,116 @@
|
||||
from lektor.builder import Builder, PathCache
|
||||
from lektor.db import Record
|
||||
from lektor.sourceobj import SourceObject
|
||||
from lektor.utils import build_url
|
||||
|
||||
from typing import Set, Dict, List, Optional, Tuple
|
||||
from .vobj import GroupBySource
|
||||
from .config import Config, AnyConfig
|
||||
from .watcher import Watcher
|
||||
|
||||
|
||||
class GroupBy:
|
||||
'''
|
||||
Process all children with matching conditions under specified page.
|
||||
Creates a grouping of pages with similar (self-defined) attributes.
|
||||
The grouping is performed only once per build.
|
||||
'''
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._watcher = [] # type: List[Watcher]
|
||||
self._results = [] # type: List[GroupBySource]
|
||||
self._resolver = {} # type: Dict[str, Tuple[str, Config]]
|
||||
|
||||
# ----------------
|
||||
# Add observer
|
||||
# ----------------
|
||||
|
||||
def add_watcher(self, key: str, config: AnyConfig) -> Watcher:
|
||||
''' Init Config and add to watch list. '''
|
||||
w = Watcher(Config.from_any(key, config))
|
||||
self._watcher.append(w)
|
||||
return w
|
||||
|
||||
# -----------
|
||||
# Builder
|
||||
# -----------
|
||||
|
||||
def clear_previous_results(self) -> None:
|
||||
''' Reset prvious results. Must be called before each build. '''
|
||||
self._watcher.clear()
|
||||
self._results.clear()
|
||||
self._resolver.clear()
|
||||
|
||||
def get_dependencies(self) -> Set[str]:
|
||||
deps = set() # type: Set[str]
|
||||
for w in self._watcher:
|
||||
deps.update(w.config.dependencies)
|
||||
return deps
|
||||
|
||||
def make_cluster(self, builder: Builder) -> None:
|
||||
''' Iterate over all children and perform groupby. '''
|
||||
# remove disabled watchers
|
||||
self._watcher = [w for w in self._watcher if w.config.enabled]
|
||||
if not self._watcher:
|
||||
return
|
||||
# initialize remaining (enabled) watchers
|
||||
for w in self._watcher:
|
||||
w.initialize(builder.pad.db)
|
||||
# iterate over whole build tree
|
||||
queue = builder.pad.get_all_roots() # type: List[SourceObject]
|
||||
while queue:
|
||||
record = queue.pop()
|
||||
self.queue_now(record)
|
||||
if hasattr(record, 'attachments'):
|
||||
queue.extend(record.attachments) # type: ignore[attr-defined]
|
||||
if hasattr(record, 'children'):
|
||||
queue.extend(record.children) # type: ignore[attr-defined]
|
||||
# build artifacts
|
||||
for w in self._watcher:
|
||||
root = builder.pad.get(w.config.root)
|
||||
for vobj in w.iter_sources(root):
|
||||
self._results.append(vobj)
|
||||
if vobj.slug:
|
||||
self._resolver[vobj.url_path] = (vobj.group, w.config)
|
||||
self._watcher.clear()
|
||||
|
||||
def queue_now(self, node: SourceObject) -> None:
|
||||
''' Process record immediatelly (No-Op if already processed). '''
|
||||
if isinstance(node, Record):
|
||||
for w in self._watcher:
|
||||
if w.should_process(node):
|
||||
w.process(node)
|
||||
|
||||
def build_all(self, builder: Builder) -> None:
|
||||
''' Create virtual objects and build sources. '''
|
||||
path_cache = PathCache(builder.env)
|
||||
for vobj in self._results:
|
||||
if vobj.slug:
|
||||
builder.build(vobj, path_cache)
|
||||
del path_cache
|
||||
self._results.clear() # garbage collect weak refs
|
||||
|
||||
# -----------------
|
||||
# Path resolver
|
||||
# -----------------
|
||||
|
||||
def resolve_dev_server_path(self, node: SourceObject, pieces: List[str]) \
|
||||
-> Optional[GroupBySource]:
|
||||
''' Dev server only: Resolves path/ -> path/index.html '''
|
||||
if isinstance(node, Record):
|
||||
rv = self._resolver.get(build_url([node.url_path] + pieces))
|
||||
if rv:
|
||||
return GroupBySource(node, group=rv[0], config=rv[1])
|
||||
return None
|
||||
|
||||
def resolve_virtual_path(self, node: SourceObject, pieces: List[str]) \
|
||||
-> Optional[GroupBySource]:
|
||||
''' Admin UI only: Prevent server error and null-redirect. '''
|
||||
if isinstance(node, Record) and len(pieces) >= 2:
|
||||
path = node['_path'] # type: str
|
||||
key, grp, *_ = pieces
|
||||
for group, conf in self._resolver.values():
|
||||
if key == conf.key and path == conf.root:
|
||||
if conf.slugify(group) == grp:
|
||||
return GroupBySource(node, group, conf)
|
||||
return None
|
||||
67
lektor_groupby/plugin.py
Normal file
67
lektor_groupby/plugin.py
Normal file
@@ -0,0 +1,67 @@
|
||||
from lektor.builder import Builder # typing
|
||||
from lektor.pluginsystem import Plugin # subclass
|
||||
from lektor.sourceobj import SourceObject # typing
|
||||
|
||||
from typing import List, Optional, Iterator
|
||||
from .vobj import GroupBySource, GroupByBuildProgram, VPATH
|
||||
from .groupby import GroupBy
|
||||
from .pruner import prune
|
||||
from .watcher import GroupByCallbackArgs # typing
|
||||
|
||||
|
||||
class GroupByPlugin(Plugin):
|
||||
name = 'GroupBy Plugin'
|
||||
description = 'Cluster arbitrary records with field attribute keyword.'
|
||||
|
||||
def on_setup_env(self, **extra: object) -> None:
|
||||
self.creator = GroupBy()
|
||||
self.env.add_build_program(GroupBySource, GroupByBuildProgram)
|
||||
self.env.jinja_env.filters.update(vgroups=GroupBySource.of_record)
|
||||
|
||||
# resolve /tag/rss/ -> /tag/rss/index.html (local server only)
|
||||
@self.env.urlresolver
|
||||
def a(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
|
||||
return self.creator.resolve_dev_server_path(node, parts)
|
||||
|
||||
# resolve virtual objects in admin UI
|
||||
@self.env.virtualpathresolver(VPATH.lstrip('@'))
|
||||
def b(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
|
||||
return self.creator.resolve_virtual_path(node, parts)
|
||||
|
||||
def _load_quick_config(self) -> None:
|
||||
''' Load config file quick listeners. '''
|
||||
config = self.get_config()
|
||||
for key in config.sections():
|
||||
if '.' in key: # e.g., key.fields and key.key_map
|
||||
continue
|
||||
|
||||
watcher = self.creator.add_watcher(key, config)
|
||||
split = config.get(key + '.split') # type: str
|
||||
|
||||
@watcher.grouping()
|
||||
def _fn(args: GroupByCallbackArgs) -> Iterator[str]:
|
||||
val = args.field
|
||||
if isinstance(val, str):
|
||||
val = val.split(split) if split else [val] # make list
|
||||
if isinstance(val, list):
|
||||
yield from val
|
||||
|
||||
def on_before_build_all(self, builder: Builder, **extra: object) -> None:
|
||||
self.creator.clear_previous_results()
|
||||
self._load_quick_config()
|
||||
# let other plugins register their @groupby.watch functions
|
||||
self.emit('before-build-all', groupby=self.creator, builder=builder)
|
||||
self.config_dependencies = self.creator.get_dependencies()
|
||||
self.creator.make_cluster(builder)
|
||||
|
||||
def on_before_build(self, source: SourceObject, **extra: object) -> None:
|
||||
# before-build may be called before before-build-all (issue #1017)
|
||||
# make sure it is evaluated immediatelly
|
||||
self.creator.queue_now(source)
|
||||
|
||||
def on_after_build_all(self, builder: Builder, **extra: object) -> None:
|
||||
self.creator.build_all(builder)
|
||||
|
||||
def on_after_prune(self, builder: Builder, **extra: object) -> None:
|
||||
# TODO: find a better way to prune unreferenced elements
|
||||
prune(builder, VPATH)
|
||||
39
lektor_groupby/pruner.py
Normal file
39
lektor_groupby/pruner.py
Normal file
@@ -0,0 +1,39 @@
|
||||
'''
|
||||
Static collector for build-artifact urls.
|
||||
All non-tracked VPATH-urls will be pruned after build.
|
||||
'''
|
||||
from lektor.builder import Builder # typing
|
||||
from lektor.reporter import reporter # report_pruned_artifact
|
||||
from lektor.utils import prune_file_and_folder
|
||||
|
||||
_cache = set()
|
||||
# Note: this var is static or otherwise two instances of
|
||||
# this module would prune each others artifacts.
|
||||
|
||||
|
||||
def track_not_prune(url: str) -> None:
|
||||
''' Add url to build cache to prevent pruning. '''
|
||||
_cache.add(url.lstrip('/'))
|
||||
|
||||
|
||||
def prune(builder: Builder, vpath: str) -> None:
|
||||
''' Remove previously generated, unreferenced Artifacts. '''
|
||||
vpath = '@' + vpath.lstrip('@') # just in case of user error
|
||||
dest_path = builder.destination_path
|
||||
con = builder.connect_to_database()
|
||||
try:
|
||||
with builder.new_build_state() as build_state:
|
||||
for url, file in build_state.iter_artifacts():
|
||||
if url.lstrip('/') in _cache:
|
||||
continue # generated in this build-run
|
||||
infos = build_state.get_artifact_dependency_infos(url, [])
|
||||
for artifact_name, _ in infos:
|
||||
if vpath not in artifact_name:
|
||||
continue # we only care about our Virtuals
|
||||
reporter.report_pruned_artifact(url)
|
||||
prune_file_and_folder(file.filename, dest_path)
|
||||
build_state.remove_artifact(url)
|
||||
break # there is only one VPATH-entry per source
|
||||
finally:
|
||||
con.close()
|
||||
_cache.clear()
|
||||
27
lektor_groupby/util.py
Normal file
27
lektor_groupby/util.py
Normal file
@@ -0,0 +1,27 @@
|
||||
from lektor.reporter import reporter, style
|
||||
|
||||
from typing import List
|
||||
from itertools import groupby
|
||||
|
||||
|
||||
def report_config_error(key: str, field: str, val: str, e: Exception) -> None:
|
||||
''' Send error message to Lektor reporter. Indicate which field is bad. '''
|
||||
msg = '[ERROR] invalid config for [{}.{}] = "{}", Error: {}'.format(
|
||||
key, field, val, repr(e))
|
||||
try:
|
||||
reporter._write_line(style(msg, fg='red'))
|
||||
except Exception:
|
||||
print(msg) # fallback in case Lektor API changes
|
||||
|
||||
|
||||
def most_used_key(keys: List[str]) -> str:
|
||||
if len(keys) < 3:
|
||||
return keys[0] # TODO: first vs last occurrence
|
||||
best_count = 0
|
||||
best_key = ''
|
||||
for key, itr in groupby(keys):
|
||||
count = sum(1 for i in itr)
|
||||
if count > best_count: # TODO: (>) vs (>=), first vs last occurrence
|
||||
best_count = count
|
||||
best_key = key
|
||||
return best_key
|
||||
195
lektor_groupby/vobj.py
Normal file
195
lektor_groupby/vobj.py
Normal file
@@ -0,0 +1,195 @@
|
||||
from lektor.build_programs import BuildProgram # subclass
|
||||
from lektor.builder import Artifact # typing
|
||||
from lektor.context import get_ctx
|
||||
from lektor.db import Record # typing
|
||||
from lektor.environment import Expression
|
||||
from lektor.sourceobj import VirtualSourceObject # subclass
|
||||
from lektor.utils import build_url
|
||||
|
||||
from typing import Dict, List, Any, Optional, Iterator
|
||||
from weakref import WeakSet
|
||||
from .config import Config
|
||||
from .pruner import track_not_prune
|
||||
from .util import report_config_error
|
||||
|
||||
VPATH = '@groupby' # potentially unsafe. All matching entries are pruned.
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# VirtualSource
|
||||
# -----------------------------------
|
||||
|
||||
class GroupBySource(VirtualSourceObject):
|
||||
'''
|
||||
Holds information for a single group/cluster.
|
||||
This object is accessible in your template file.
|
||||
Attributes: record, key, group, slug, children, config
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
record: Record,
|
||||
group: str,
|
||||
config: Config,
|
||||
children: Optional[Dict[Record, List[Any]]] = None,
|
||||
) -> None:
|
||||
super().__init__(record)
|
||||
self.key = config.slugify(group)
|
||||
self.group = group
|
||||
self.config = config
|
||||
# evaluate slug Expression
|
||||
if config.slug and '{key}' in config.slug:
|
||||
self.slug = config.slug.replace('{key}', self.key)
|
||||
else:
|
||||
self.slug = self._eval(config.slug, field='slug')
|
||||
assert self.slug != Ellipsis, 'invalid config: ' + config.slug
|
||||
if self.slug and self.slug.endswith('/index.html'):
|
||||
self.slug = self.slug[:-10]
|
||||
# make sure children are on the same pad
|
||||
self._children = {} # type: Dict[Record, List[Any]]
|
||||
for child, extras in (children or {}).items():
|
||||
if child.pad != record.pad:
|
||||
child = record.pad.get(child.path)
|
||||
self._children[child] = extras
|
||||
self._reverse_reference_records()
|
||||
# extra fields
|
||||
for attr, expr in config.fields.items():
|
||||
setattr(self, attr, self._eval(expr, field='fields.' + attr))
|
||||
|
||||
def _eval(self, value: Any, *, field: str) -> Any:
|
||||
''' Internal only: evaluates Lektor config file field expression. '''
|
||||
if not isinstance(value, str):
|
||||
return value
|
||||
pad = self.record.pad
|
||||
alt = self.record.alt
|
||||
try:
|
||||
return Expression(pad.env, value).evaluate(pad, this=self, alt=alt)
|
||||
except Exception as e:
|
||||
report_config_error(self.config.key, field, value, e)
|
||||
return Ellipsis
|
||||
|
||||
# ---------------------
|
||||
# Lektor properties
|
||||
# ---------------------
|
||||
|
||||
@property
|
||||
def path(self) -> str:
|
||||
# Used in VirtualSourceInfo, used to prune VirtualObjects
|
||||
return f'{self.record.path}{VPATH}/{self.config.key}/{self.key}'
|
||||
|
||||
@property
|
||||
def url_path(self) -> str:
|
||||
# Actual path to resource as seen by the browser
|
||||
return build_url([self.record.path, self.slug]) # slug can be None!
|
||||
|
||||
def iter_source_filenames(self) -> Iterator[str]:
|
||||
''' Enumerate all dependencies '''
|
||||
if self.config.dependencies:
|
||||
yield from self.config.dependencies
|
||||
for record in self._children:
|
||||
yield from record.iter_source_filenames()
|
||||
|
||||
# -----------------------
|
||||
# Properties & Helper
|
||||
# -----------------------
|
||||
|
||||
@property
|
||||
def children(self) -> Dict[Record, List[Any]]:
|
||||
''' Returns dict with page record key and (optional) extra value. '''
|
||||
return self._children
|
||||
|
||||
@property
|
||||
def first_child(self) -> Optional[Record]:
|
||||
''' Returns first referencing page record. '''
|
||||
if self._children:
|
||||
return iter(self._children).__next__()
|
||||
return None
|
||||
|
||||
@property
|
||||
def first_extra(self) -> Optional[Any]:
|
||||
''' Returns first additional / extra info object of first page. '''
|
||||
if not self._children:
|
||||
return None
|
||||
val = iter(self._children.values()).__next__()
|
||||
return val[0] if val else None
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
# Used for virtual path resolver
|
||||
if key in ('_path', '_alt'):
|
||||
return getattr(self, key[1:])
|
||||
return self.__missing__(key) # type: ignore[attr-defined]
|
||||
|
||||
def __lt__(self, other: 'GroupBySource') -> bool:
|
||||
# Used for |sort filter ("group" is the provided original string)
|
||||
return self.group < other.group
|
||||
|
||||
def __eq__(self, other: object) -> bool:
|
||||
# Used for |unique filter
|
||||
if self is other:
|
||||
return True
|
||||
return isinstance(other, GroupBySource) and \
|
||||
self.path == other.path and self.slug == other.slug
|
||||
|
||||
def __hash__(self) -> int:
|
||||
# Used for hashing in set and dict
|
||||
return hash((self.path, self.slug))
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return '<GroupBySource path="{}" children={}>'.format(
|
||||
self.path, len(self._children))
|
||||
|
||||
# ---------------------
|
||||
# Reverse Reference
|
||||
# ---------------------
|
||||
|
||||
def _reverse_reference_records(self) -> None:
|
||||
''' Attach self to page records. '''
|
||||
for child in self._children:
|
||||
if not hasattr(child, '_vgroups'):
|
||||
child._vgroups = WeakSet() # type: ignore[attr-defined]
|
||||
child._vgroups.add(self) # type: ignore[attr-defined]
|
||||
|
||||
@staticmethod
|
||||
def of_record(
|
||||
record: Record,
|
||||
*keys: str,
|
||||
recursive: bool = False
|
||||
) -> Iterator['GroupBySource']:
|
||||
''' Extract all referencing groupby virtual objects from a page. '''
|
||||
ctx = get_ctx()
|
||||
# manage dependencies
|
||||
if ctx:
|
||||
for dep in ctx.env.plugins['groupby'].config_dependencies:
|
||||
ctx.record_dependency(dep)
|
||||
# find groups
|
||||
proc_list = [record]
|
||||
while proc_list:
|
||||
page = proc_list.pop(0)
|
||||
if recursive and hasattr(page, 'children'):
|
||||
proc_list.extend(page.children) # type: ignore[attr-defined]
|
||||
if not hasattr(page, '_vgroups'):
|
||||
continue
|
||||
for vobj in page._vgroups: # type: ignore[attr-defined]
|
||||
if not keys or vobj.config.key in keys:
|
||||
yield vobj
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# BuildProgram
|
||||
# -----------------------------------
|
||||
|
||||
class GroupByBuildProgram(BuildProgram):
|
||||
''' Generate Build-Artifacts and write files. '''
|
||||
|
||||
def produce_artifacts(self) -> None:
|
||||
url = self.source.url_path
|
||||
if url.endswith('/'):
|
||||
url += 'index.html'
|
||||
self.declare_artifact(url, sources=list(
|
||||
self.source.iter_source_filenames()))
|
||||
track_not_prune(url)
|
||||
|
||||
def build_artifact(self, artifact: Artifact) -> None:
|
||||
get_ctx().record_virtual_dependency(self.source)
|
||||
artifact.render_template_into(
|
||||
self.source.config.template, this=self.source)
|
||||
192
lektor_groupby/watcher.py
Normal file
192
lektor_groupby/watcher.py
Normal file
@@ -0,0 +1,192 @@
|
||||
from lektor.db import Database, Record # typing
|
||||
from lektor.types.flow import Flow, FlowType
|
||||
from lektor.utils import bool_from_string
|
||||
|
||||
from typing import Set, Dict, List, Tuple, Any, Union, NamedTuple
|
||||
from typing import Optional, Callable, Iterator, Generator
|
||||
from .vobj import GroupBySource
|
||||
from .config import Config
|
||||
from .util import most_used_key
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Typing
|
||||
# -----------------------------------
|
||||
|
||||
class FieldKeyPath(NamedTuple):
|
||||
fieldKey: str
|
||||
flowIndex: Optional[int] = None
|
||||
flowKey: Optional[str] = None
|
||||
|
||||
|
||||
class GroupByCallbackArgs(NamedTuple):
|
||||
record: Record
|
||||
key: FieldKeyPath
|
||||
field: Any # lektor model data-field value
|
||||
|
||||
|
||||
GroupingCallback = Callable[[GroupByCallbackArgs], Union[
|
||||
Iterator[Union[str, Tuple[str, Any]]],
|
||||
Generator[Union[str, Tuple[str, Any]], Optional[str], None],
|
||||
]]
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# ModelReader
|
||||
# -----------------------------------
|
||||
|
||||
class GroupByModelReader:
|
||||
''' Find models and flow-models which contain attribute '''
|
||||
|
||||
def __init__(self, db: Database, attrib: str) -> None:
|
||||
self._flows = {} # type: Dict[str, Set[str]]
|
||||
self._models = {} # type: Dict[str, Dict[str, str]]
|
||||
# find flow blocks containing attribute
|
||||
for key, flow in db.flowblocks.items():
|
||||
tmp1 = set(f.name for f in flow.fields
|
||||
if bool_from_string(f.options.get(attrib, False)))
|
||||
if tmp1:
|
||||
self._flows[key] = tmp1
|
||||
# find models and flow-blocks containing attribute
|
||||
for key, model in db.datamodels.items():
|
||||
tmp2 = {} # Dict[str, str]
|
||||
for field in model.fields:
|
||||
if bool_from_string(field.options.get(attrib, False)):
|
||||
tmp2[field.name] = '*' # include all children
|
||||
elif isinstance(field.type, FlowType) and self._flows:
|
||||
# only processed if at least one flow has attrib
|
||||
fbs = field.type.flow_blocks
|
||||
# if fbs == None, all flow-blocks are allowed
|
||||
if fbs is None or any(x in self._flows for x in fbs):
|
||||
tmp2[field.name] = '?' # only some flow blocks
|
||||
if tmp2:
|
||||
self._models[key] = tmp2
|
||||
|
||||
def read(
|
||||
self,
|
||||
record: Record,
|
||||
flatten: bool = False
|
||||
) -> Iterator[Tuple[FieldKeyPath, Any]]:
|
||||
'''
|
||||
Enumerate all fields of a Record with attrib = True.
|
||||
Flows are either returned directly (flatten=False) or
|
||||
expanded so that each flow-block is yielded (flatten=True)
|
||||
'''
|
||||
assert isinstance(record, Record)
|
||||
for r_key, subs in self._models.get(record.datamodel.id, {}).items():
|
||||
field = record[r_key]
|
||||
if not field:
|
||||
continue
|
||||
if subs == '*': # either normal field or flow type (all blocks)
|
||||
if flatten and isinstance(field, Flow):
|
||||
for i, flow in enumerate(field.blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key, block in flow._data.items():
|
||||
if f_key.startswith('_'): # e.g., _flowblock
|
||||
continue
|
||||
yield FieldKeyPath(r_key, i, f_key), block
|
||||
else:
|
||||
yield FieldKeyPath(r_key), field
|
||||
else: # always flow type (only some blocks)
|
||||
for i, flow in enumerate(field.blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key in self._flows.get(flowtype, []):
|
||||
yield FieldKeyPath(r_key, i, f_key), flow[f_key]
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Watcher
|
||||
# -----------------------------------
|
||||
|
||||
class Watcher:
|
||||
'''
|
||||
Callback is called with (Record, FieldKeyPath, field-value).
|
||||
Callback may yield one or more (group, extra-info) tuples.
|
||||
'''
|
||||
|
||||
def __init__(self, config: Config) -> None:
|
||||
self.config = config
|
||||
self.flatten = True
|
||||
self.callback = None # type: GroupingCallback #type:ignore[assignment]
|
||||
|
||||
def grouping(self, flatten: bool = True) \
|
||||
-> Callable[[GroupingCallback], None]:
|
||||
'''
|
||||
Decorator to subscribe to attrib-elements.
|
||||
If flatten = False, dont explode FlowType.
|
||||
|
||||
(record, field-key, field) -> (group, extra-info)
|
||||
'''
|
||||
def _decorator(fn: GroupingCallback) -> None:
|
||||
self.flatten = flatten
|
||||
self.callback = fn
|
||||
return _decorator
|
||||
|
||||
def initialize(self, db: Database) -> None:
|
||||
''' Reset internal state. You must initialize before each build! '''
|
||||
assert callable(self.callback), 'No grouping callback provided.'
|
||||
self._root = self.config.root
|
||||
self._model_reader = GroupByModelReader(db, attrib=self.config.key)
|
||||
self._state = {} # type: Dict[str, Dict[Record, List[Any]]]
|
||||
self._group_map = {} # type: Dict[str, List[str]]
|
||||
self._processed = set() # type: Set[str]
|
||||
|
||||
def should_process(self, node: Record) -> bool:
|
||||
''' Check if record path is being watched. '''
|
||||
return node['_path'].startswith(self._root)
|
||||
|
||||
def process(self, record: Record) -> None:
|
||||
'''
|
||||
Will iterate over all record fields and call the callback method.
|
||||
Each record is guaranteed to be processed only once.
|
||||
'''
|
||||
if record.path in self._processed:
|
||||
return
|
||||
self._processed.add(record.path)
|
||||
for key, field in self._model_reader.read(record, self.flatten):
|
||||
_gen = self.callback(GroupByCallbackArgs(record, key, field))
|
||||
try:
|
||||
obj = next(_gen)
|
||||
while True:
|
||||
if not isinstance(obj, (str, tuple)):
|
||||
raise TypeError(f'Unsupported groupby yield: {obj}')
|
||||
slug = self._persist(record, obj)
|
||||
# return slugified group key and continue iteration
|
||||
if isinstance(_gen, Generator) and not _gen.gi_yieldfrom:
|
||||
obj = _gen.send(slug)
|
||||
else:
|
||||
obj = next(_gen)
|
||||
except StopIteration:
|
||||
del _gen
|
||||
|
||||
def _persist(self, record: Record, obj: Union[str, tuple]) -> str:
|
||||
group = obj if isinstance(obj, str) else obj[0]
|
||||
slug = self.config.slugify(group)
|
||||
# init group-key
|
||||
if slug not in self._state:
|
||||
self._state[slug] = {}
|
||||
self._group_map[slug] = []
|
||||
# _group_map is later used to find most used group
|
||||
self._group_map[slug].append(group)
|
||||
# init group extras
|
||||
if record not in self._state[slug]:
|
||||
self._state[slug][record] = []
|
||||
# (optional) append extra
|
||||
if isinstance(obj, tuple):
|
||||
self._state[slug][record].append(obj[1])
|
||||
return slug
|
||||
|
||||
def iter_sources(self, root: Record) -> Iterator[GroupBySource]:
|
||||
''' Prepare and yield GroupBySource elements. '''
|
||||
for key, children in self._state.items():
|
||||
group = most_used_key(self._group_map[key])
|
||||
yield GroupBySource(root, group, self.config, children=children)
|
||||
# cleanup. remove this code if you'd like to iter twice
|
||||
del self._model_reader
|
||||
del self._state
|
||||
del self._group_map
|
||||
del self._processed
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return '<GroupByWatcher key="{}" enabled={} callback={}>'.format(
|
||||
self.config.key, self.config.enabled, self.callback)
|
||||
3
setup.py
3
setup.py
@@ -13,7 +13,7 @@ setup(
|
||||
},
|
||||
author='relikd',
|
||||
url='https://github.com/relikd/lektor-groupby-plugin',
|
||||
version='0.9.2',
|
||||
version='0.9.5',
|
||||
description='Cluster arbitrary records with field attribute keyword.',
|
||||
long_description=longdesc,
|
||||
long_description_content_type="text/markdown",
|
||||
@@ -27,7 +27,6 @@ setup(
|
||||
'cluster',
|
||||
],
|
||||
classifiers=[
|
||||
'Development Status :: 5 - Production/Stable',
|
||||
'Environment :: Web Environment',
|
||||
'Environment :: Plugins',
|
||||
'Framework :: Lektor',
|
||||
|
||||
Reference in New Issue
Block a user