Compare commits
50 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
227c4cdac9 | ||
|
|
3139b5205a | ||
|
|
f32046dffb | ||
|
|
85df707d63 | ||
|
|
7582029abf | ||
|
|
fb9a690f79 | ||
|
|
491c06e22f | ||
|
|
7d668892a6 | ||
|
|
4b63fae4d6 | ||
|
|
521ac39a83 | ||
|
|
390d44a02c | ||
|
|
7c324e5909 | ||
|
|
0891be06e2 | ||
|
|
e7ae59fadf | ||
|
|
b75102a211 | ||
|
|
7c98d74875 | ||
|
|
3e60e536f5 | ||
|
|
d58529f4cc | ||
|
|
03475e3e5a | ||
|
|
5387256b93 | ||
|
|
e67489ab0b | ||
|
|
8e250fb665 | ||
|
|
a0b53c7566 | ||
|
|
f13bd3dfc6 | ||
|
|
fb8321744e | ||
|
|
eb0a60ab33 | ||
|
|
c149831808 | ||
|
|
7f28c53107 | ||
|
|
5118d19532 | ||
|
|
1d9629566c | ||
|
|
8ae5376d41 | ||
|
|
340bc6611b | ||
|
|
9dcd704283 | ||
|
|
d689a6cdf7 | ||
|
|
b05dd31ff0 | ||
|
|
16a26afdce | ||
|
|
c618ee458b | ||
|
|
55916a4519 | ||
|
|
a694149d04 | ||
|
|
831cfa4e9c | ||
|
|
298e0d4a62 | ||
|
|
2a6bdf05fd | ||
|
|
df4be7c60a | ||
|
|
637524a615 | ||
|
|
a6d9f715f9 | ||
|
|
d6df547682 | ||
|
|
ebc29459ec | ||
|
|
adb26e343e | ||
|
|
97b40b4886 | ||
|
|
479ff9b964 |
2
.gitignore
vendored
2
.gitignore
vendored
@@ -1,5 +1,5 @@
|
||||
.DS_Store
|
||||
/dist-env/
|
||||
/env-publish/
|
||||
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
|
||||
155
CHANGELOG.md
Normal file
155
CHANGELOG.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Changelog
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/0.9.8/),
|
||||
and this project does adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
|
||||
## [0.9.8] – 2022-12-20
|
||||
|
||||
### Added
|
||||
- Support for Alternatives
|
||||
- Support for Pagination
|
||||
- Support for additional yield types (str, int, float, bool)
|
||||
- Support for sorting `GroupBySource` children
|
||||
- Support for sorting `vgroups` filter
|
||||
- Config option `.replace_none_key` to replace `None` with another value
|
||||
- Config option `.key_obj_fn` (function) can be used to map complex objects to simple values (e.g., list of strings -> count as int). In your jinja template you may use `X` (the object) and `ARGS` (the `GroupByCallbackArgs`).
|
||||
- New property `supports_pagination` (bool) for `GroupBySource`
|
||||
- Partial building. Only process `Watcher` which are used during template rendering.
|
||||
- Rebuild `GroupBySource` only once after a `Record` update
|
||||
|
||||
### Changed
|
||||
- Use `Query` for children instead of `Record` list
|
||||
- Rename `GroupBySource.group` to `GroupBySource.key_obj`
|
||||
- Yield return `GroupBySource` during `watcher.grouping()` instead of slugified key
|
||||
- Postpone `Record` processing until `make_once()`
|
||||
- Allow preprocessing with `pre_build=True` as optional parameter for `groupby.add_watcher()` (useful for modifying source before build)
|
||||
- Evaluate `fields` attributes upon access, not initialization (this comes with a more fine-grained dependency tracking)
|
||||
- Resolver groups virtual pages per groupby config key (before it was just a list of all groupby sources mixed together)
|
||||
- Refactor pruning by adding a `VirtualPruner` vobj
|
||||
- Pruning is performed directly on the database
|
||||
- `GroupBySource.path` may include a page number suffix `/2`
|
||||
- `GroupBySource.url_path` may include a page number and custom `url_suffix`
|
||||
|
||||
### Removed
|
||||
- `GroupingCallback` may no longer yield an extra object. The usage was cumbersome and can be replaced with the `.fields` config option.
|
||||
|
||||
### Fixed
|
||||
- `GroupBySource` not updated on template edit
|
||||
- `most_used_key` with empty list
|
||||
- Don't throw exception if `GroupBySource` is printed before finalize
|
||||
- Hotfix for Lektor issue #1085 by avoiding `TypeError`
|
||||
- Add missing dependencies during `vgroups` filter
|
||||
- Include model-fields with null value on yield
|
||||
|
||||
|
||||
|
||||
## [0.9.7] – 2022-04-22
|
||||
|
||||
### Changed
|
||||
- Refactor `GroupBySource` init method
|
||||
- Decouple `fields` expression processing from init
|
||||
|
||||
### Fixed
|
||||
- Keep order of groups intact
|
||||
|
||||
|
||||
|
||||
## [0.9.6] – 2022-04-13
|
||||
|
||||
### Added
|
||||
- Set extra-info default to the model-key that generated the group.
|
||||
- Reuse previously declared `fields` attributes in later `fields`.
|
||||
|
||||
### Changed
|
||||
- Thread-safe building. Each groupby is performed on the builder which initiated the build.
|
||||
- Deferred building. The groupby callback is only called when it is accessed for the first time.
|
||||
- Build-on-access. If there are no changes, no groupby build is performed.
|
||||
|
||||
### Fixed
|
||||
- Inconsistent behavior due to concurrent building (see above)
|
||||
- Case insensitive default group sort
|
||||
- Using the split config-option now trims whitespace
|
||||
- `most_used_key` working properly
|
||||
|
||||
|
||||
|
||||
## [0.9.5] – 2022-04-07
|
||||
|
||||
### Fixed
|
||||
- Allow model instances without flow-blocks
|
||||
|
||||
|
||||
|
||||
## [0.9.4] – 2022-04-06
|
||||
|
||||
### Fixed
|
||||
- Error handling for GroupBySource `__getitem__` by raising `__missing__`
|
||||
- Reuse GroupBySource if two group names result in the same slug
|
||||
|
||||
|
||||
|
||||
## [0.9.3] – 2022-04-06
|
||||
|
||||
### Added
|
||||
- Config option `.fields` can add arbitrary attributes to the groupby
|
||||
- Config option `.key_map` allows to replace keys with other values (e.g., "C#" -> "C-Sharp")
|
||||
- Set `slug = None` to prevent rendering of groupby pages
|
||||
- Query groupby of children
|
||||
|
||||
### Changed
|
||||
- Another full refactoring, constantly changing, everything is different ... again
|
||||
|
||||
|
||||
|
||||
## [0.9.2] – 2022-04-01
|
||||
|
||||
### Fixed
|
||||
- Prevent duplicate processing of records
|
||||
|
||||
|
||||
|
||||
## [0.9.1] – 2022-03-31
|
||||
|
||||
### Added
|
||||
- Example project
|
||||
- Before- and after-init hooks
|
||||
- More type hints (incl. bugfixes)
|
||||
|
||||
### Changed
|
||||
- Encapsulate logic into separate classes
|
||||
|
||||
### Fixed
|
||||
- Concurrency issues by complete refactoring
|
||||
- Virtual path and remove virtual path resolver
|
||||
|
||||
|
||||
|
||||
## [0.9] – 2022-03-27
|
||||
|
||||
### Fixed
|
||||
- Groupby is now generated before main page
|
||||
- PyPi readme
|
||||
|
||||
|
||||
|
||||
## [0.8] – 2022-03-25
|
||||
|
||||
Initial release
|
||||
|
||||
|
||||
[Unreleased]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.8...HEAD
|
||||
[0.9.8]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.7...v0.9.8
|
||||
[0.9.7]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.6...v0.9.7
|
||||
[0.9.6]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.5...v0.9.6
|
||||
[0.9.5]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.4...v0.9.5
|
||||
[0.9.4]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.3...v0.9.4
|
||||
[0.9.3]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.2...v0.9.3
|
||||
[0.9.2]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9.1...v0.9.2
|
||||
[0.9.1]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.9...v0.9.1
|
||||
[0.9]: https://github.com/relikd/lektor-groupby-plugin/compare/v0.8...v0.9
|
||||
[0.8]: https://github.com/relikd/lektor-groupby-plugin/releases/tag/v0.8
|
||||
28
Makefile
28
Makefile
@@ -1,21 +1,17 @@
|
||||
.PHONY: help
|
||||
help:
|
||||
@echo 'commands:'
|
||||
@echo ' dist'
|
||||
|
||||
dist-env:
|
||||
@echo Creating virtual environment...
|
||||
@python3 -m venv 'dist-env'
|
||||
@source dist-env/bin/activate && pip install twine
|
||||
|
||||
.PHONY: dist
|
||||
dist: dist-env
|
||||
dist: setup.py lektor_groupby/*
|
||||
[ -z "$${VIRTUAL_ENV}" ] # you can not do this inside a virtual environment.
|
||||
rm -rf dist
|
||||
@echo Building...
|
||||
python3 setup.py sdist bdist_wheel
|
||||
@echo
|
||||
rm -rf ./*.egg-info/ ./build/ MANIFEST
|
||||
|
||||
env-publish:
|
||||
@echo Creating virtual environment...
|
||||
@python3 -m venv 'env-publish'
|
||||
@source env-publish/bin/activate && pip install twine
|
||||
|
||||
.PHONY: publish
|
||||
publish: dist env-publish
|
||||
[ -z "$${VIRTUAL_ENV}" ] # you can not do this inside a virtual environment.
|
||||
@echo Publishing...
|
||||
@echo "\033[0;31mEnter your PyPI token:\033[0m"
|
||||
@source dist-env/bin/activate && export TWINE_USERNAME='__token__' && twine upload dist/*
|
||||
@echo "\033[0;31mEnter PyPI token in password prompt:\033[0m"
|
||||
@source env-publish/bin/activate && export TWINE_USERNAME='__token__' && twine upload dist/*
|
||||
|
||||
@@ -5,11 +5,10 @@ Can be used for tagging or similar tasks.
|
||||
The grouping algorithm is performed once.
|
||||
Contrary to, at least, cubic runtime if doing the same with Pad queries.
|
||||
|
||||
To install this plugin, modify your Lektor project file:
|
||||
Install this plugin or modify your Lektor project file:
|
||||
|
||||
```ini
|
||||
[packages]
|
||||
lektor-groupby = 0.9.1
|
||||
```sh
|
||||
lektor plugin add groupby
|
||||
```
|
||||
|
||||
Optionally, enable a basic config:
|
||||
@@ -17,7 +16,7 @@ Optionally, enable a basic config:
|
||||
```ini
|
||||
[tags]
|
||||
root = /
|
||||
slug = tag/{group}.html
|
||||
slug = tag/{key}.html
|
||||
template = tag.html
|
||||
split = ' '
|
||||
```
|
||||
|
||||
@@ -2,4 +2,4 @@
|
||||
name = GroupBy Examples
|
||||
|
||||
[packages]
|
||||
lektor-groupby = 0.9.1
|
||||
lektor-groupby = 0.9.8
|
||||
|
||||
7
examples/Makefile
Normal file
7
examples/Makefile
Normal file
@@ -0,0 +1,7 @@
|
||||
.PHONY: server clean plugins
|
||||
server:
|
||||
lektor server
|
||||
clean:
|
||||
lektor clean --yes -v
|
||||
plugins:
|
||||
lektor plugins flush-cache && lektor plugins list
|
||||
@@ -1,16 +1,19 @@
|
||||
# Usage
|
||||
|
||||
Overview:
|
||||
- the [quick config](#quick-config) example shows how you can use the plugin config to setup a quick and easy tagging system.
|
||||
- the [simple example](#simple-example) goes into detail how this plugin works.
|
||||
- the [advanced example](#advanced-example) touches on the potentials of the plugin.
|
||||
- [quick config example](#quick-config) shows how you can use the plugin config to setup a quick and easy tagging system.
|
||||
- [simple example](#simple-example) goes into detail how to use it in your own plugin.
|
||||
- [advanced example](#advanced-example) touches on the potentials of the plugin.
|
||||
- [Misc](#misc) shows other use-cases.
|
||||
|
||||
After reading this tutorial, have a look at other plugins that use `lektor-groupby`:
|
||||
- [lektor-inlinetags](https://github.com/relikd/lektor-inlinetags-plugin)
|
||||
|
||||
|
||||
## About
|
||||
|
||||
To use the groupby plugin you have to add an attribute to your model file.
|
||||
In our case you can refer to the `models/page.ini` model:
|
||||
For this tutorial you can refer to the [`models/page.ini`](./models/page.ini) model:
|
||||
|
||||
```ini
|
||||
[fields.tags]
|
||||
@@ -25,7 +28,7 @@ type = markdown
|
||||
testC = true
|
||||
```
|
||||
|
||||
We did define three custom attributes `testA`, `testB`, and `testC`.
|
||||
We define three custom attributes `testA`, `testB`, and `testC`.
|
||||
You may add custom attributes to all of the fields.
|
||||
It is crucial that the value of the custom attribute is set to true.
|
||||
The attribute name is later used for grouping.
|
||||
@@ -35,19 +38,36 @@ The attribute name is later used for grouping.
|
||||
## Quick config
|
||||
|
||||
Relevant files:
|
||||
```
|
||||
configs/groupby.ini
|
||||
templates/example-config.html
|
||||
```
|
||||
|
||||
- [`configs/groupby.ini`](./configs/groupby.ini)
|
||||
- [`templates/example-config.html`](./templates/example-config.html)
|
||||
|
||||
|
||||
The easiest way to add tags to your site is by defining the `groupby.ini` config file.
|
||||
|
||||
```ini
|
||||
[testA]
|
||||
root = /
|
||||
slug = config/{group}.html
|
||||
slug = config/{key}.html
|
||||
template = example-config.html
|
||||
split = ' '
|
||||
enabled = True
|
||||
key_obj_fn = (X.upper() ~ ARGS.key.fieldKey) if X else 'empty'
|
||||
replace_none_key = unknown
|
||||
|
||||
[testA.children]
|
||||
order_by = -title, body
|
||||
|
||||
[testA.pagination]
|
||||
enabled = true
|
||||
per_page = 5
|
||||
url_suffix = .page.
|
||||
|
||||
[testA.fields]
|
||||
title = "Tagged: " ~ this.key_obj
|
||||
|
||||
[testA.key_map]
|
||||
Blog = News
|
||||
```
|
||||
|
||||
The configuration parameter are:
|
||||
@@ -57,9 +77,11 @@ The configuration parameter are:
|
||||
All results will be placed under this directory, e.g., `/tags/tagname/`.
|
||||
If you use `root = /blog`, the results path will be `/blog/tags/tagname/`.
|
||||
The groupby plugin will traverse all sub-pages wich contain the attribute `testA`.
|
||||
3. The `slug` parameter (`config/{group}.html`) is where the results are placed.
|
||||
3. The `slug` parameter (`config/{key}.html`) is where the results are placed.
|
||||
In our case, the path resolves to `config/tagname.html`.
|
||||
The default value is `{attrib}/{group}/index.html` which would resolve to `testA/tagname/index.html`.
|
||||
The default value is `{attrib}/{key}/index.html` which would resolve to `testA/tagname/index.html`.
|
||||
If this field contains `{key}`, it just replaces the value with the group-key.
|
||||
In all other cases the field value is evaluated in a jinja context.
|
||||
4. The `template`parameter (`example-config.html`) is used to render the results page.
|
||||
If no explicit template is set, the default template `groupby-testA.html` will be used.
|
||||
Where `testA` is replaced with whatever attribute you chose.
|
||||
@@ -68,18 +90,68 @@ The configuration parameter are:
|
||||
The split is only relevant for fields of type `string` or `text`.
|
||||
These single-line fields are then expanded to lists as well.
|
||||
If you do not provide the `split` option, the whole field value will be used as tagname.
|
||||
6. The `enabled` parameter allows you to quickly disable the grouping.
|
||||
7. The `key_obj_fn` parameter (jinja2) accepts any function-like snippet or function call.
|
||||
The context provides two variables, `X` and `ARGS`.
|
||||
The former is the raw value of the grouping, this may be a text field, markdown, or whatever custom type you have provided.
|
||||
The latter is a named tuple with `record`, `key`, and `field` values (see [simple example](#simple-example)).
|
||||
8. The `replace_none_key` parameter (string) is applied after `key_obj_fn` (if provided) and maps empty values to a default value.
|
||||
|
||||
|
||||
You can have multiple listeners, e.g., one for `/blog/` and another for `/projects/`.
|
||||
Just create as many custom attributes as you like, each having its own section.
|
||||
Just create as many custom attributes as you like, each having its own section (and subsections).
|
||||
|
||||
In your template file you have access to the children (pages) and their tags.
|
||||
The emitted `extras` for the child is a list of original tagnames.
|
||||
The `.children` subsection currently has a single config field: `order_by`.
|
||||
The usual [order-by](https://www.getlektor.com/docs/guides/page-order/) rules apply (comma separated list of keys with `-` for reversed order).
|
||||
The order-by key can be anything of the page attributes of the children.
|
||||
|
||||
The `.pagination` subsection accepts the same configuration options as the Lektor pagination [model](https://www.getlektor.com/docs/models/children/#pagination) and [guide](https://www.getlektor.com/docs/guides/pagination/).
|
||||
Plus, an additional `url_suffix` parameter if you would like to customize the URL scheme.
|
||||
|
||||
The `.fields` subsection is a list of key-value pairs which will be added as attributes to your grouping.
|
||||
You can access them in your template (e.g., `{{this.title}}`).
|
||||
All of the `.fields` values are evaluted in a jinja context, so be cautious when using plain strings.
|
||||
Further, they are evaluated on access and not on define.
|
||||
|
||||
The built-in field attributes are:
|
||||
|
||||
- `key_obj`: model returned object, e.g., "A Title?"
|
||||
- `key`: slugified value of `key_obj`, e.g., "a-title"
|
||||
- `record`: parent node, e.g., `Page(path="/")`
|
||||
- `slug`: url path under parent node, e.g. "config/a-title.html" (can be `None`)
|
||||
- `children`: the elements of the grouping (a `Query` of `Record` type)
|
||||
- `config`: configuration object (see below)
|
||||
|
||||
Without any changes, the `key` value will just be `slugify(key_obj)`.
|
||||
However, the `.key_map` subsection will replace `key_obj` with whatever replacement value is provided in the `.key_map` and then slugify.
|
||||
You could, for example, add a `C# = c-sharp` mapping, which would otherwise just be slugified to `c`.
|
||||
This is equivalent to `slugify(key_map.get(key_obj))`.
|
||||
|
||||
The `config` attribute contains the values that created the group:
|
||||
|
||||
- `key`: attribute key, e.g., `TestA`
|
||||
- `root`: as provided by init, e.g., `/`
|
||||
- `slug`: the raw value, e.g., `config/{key}.html`
|
||||
- `template`: as provided by init, e.g., `example-config.html`
|
||||
- `key_obj_fn`: as provided by init, e.g., `X.upper() if X else 'empty'`
|
||||
- `replace_none_key`: as provided by init, e.g., `unknown`
|
||||
- `enabled`: boolean
|
||||
- `dependencies`: path to config file (if initialized from config)
|
||||
- `fields`: raw values from `TestA.fields`
|
||||
- `key_map`: raw values from `TestA.key_map`
|
||||
- `pagination`: raw values from `TestA.pagination`
|
||||
- `order_by`: list of key-strings from `TestA.children.order_by`
|
||||
|
||||
In your template file you have access to the config, attributes, fields, and children (Pages):
|
||||
|
||||
```jinja2
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<li>Page: {{ child.path }}, Tags: {{ extras }}</li>
|
||||
<h2>{{ this.title }}</h2>
|
||||
<p>Key: {{ this.key }}, Attribute: {{ this.config.key }}</p>
|
||||
<ul>
|
||||
{%- for child in this.children %}
|
||||
<li>Page: {{ child.path }}</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
```
|
||||
|
||||
|
||||
@@ -87,66 +159,69 @@ The emitted `extras` for the child is a list of original tagnames.
|
||||
## Simple example
|
||||
|
||||
Relevant files:
|
||||
```
|
||||
packages/simple-example/lektor_simple.py
|
||||
templates/example-simple.html
|
||||
```
|
||||
|
||||
- [`packages/simple-example/lektor_simple.py`](./packages/simple-example/lektor_simple.py)
|
||||
- [`templates/example-simple.html`](./templates/example-simple.html)
|
||||
|
||||
|
||||
```python
|
||||
def on_groupby_after_build_all(self, groupby, builder, **extra):
|
||||
@groupby.watch('/blog', 'testB', slug='simple/{group}/index.html',
|
||||
template='example-simple.html', flatten=True)
|
||||
def convert_simple_example(args):
|
||||
value = args.field # list, since model is 'strings' type
|
||||
for tag in value:
|
||||
yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
|
||||
def on_groupby_before_build_all(self, groupby, builder, **extra):
|
||||
watcher = groupby.add_watcher('testB', {
|
||||
'root': '/blog',
|
||||
'slug': 'simple/{key}/index.html',
|
||||
'template': 'example-simple.html',
|
||||
})
|
||||
watcher.config.set_key_map({'Foo': 'bar'})
|
||||
watcher.config.set_fields({'date': datetime.now()})
|
||||
|
||||
# page = args.record # extract additional info from source
|
||||
# fieldKey, flowIndex, flowKey = args.key # or get field index
|
||||
# if flowIndex is None:
|
||||
# obj = page[fieldKey]
|
||||
# else:
|
||||
# obj = page[fieldKey].blocks[flowIndex].get(flowKey)
|
||||
@watcher.grouping(flatten=True)
|
||||
def convert_simple_example(args):
|
||||
# Yield groups
|
||||
value = args.field # type: list # since model is 'strings' type
|
||||
for tag in value:
|
||||
yield tag
|
||||
```
|
||||
|
||||
This example is roughly equivalent to the config file example.
|
||||
The parameters of the `@groupby.watch` function (`root`, `attribute`, `slug`, `template`) correspond to the same config parameters described above.
|
||||
There is a new `flatten` parameter:
|
||||
This example is roughly equivalent to the config example above – the parameters of the `groupby.add_watcher` function correspond to the same config parameters.
|
||||
Additionally, you can set other types in `set_fields` (all strings are evaluated in jinja context!).
|
||||
Refer to `lektor_simple.py` for all available configuration options.
|
||||
|
||||
- Flatten determines how Flow elements are processed.
|
||||
The `@watcher.grouping` callback generates all groups for a single watcher-attribute.
|
||||
The callback body **can** produce groupings but does not have to.
|
||||
If you choose to produce an entry, you have to `yield` a grouping object (string, int, bool, float, or object).
|
||||
In any case, `key_obj` is slugified (see above) and then used to combine & cluster pages.
|
||||
You can yield more than one entry per source.
|
||||
Or ignore pages if you don't yield anything.
|
||||
|
||||
The `@watcher.grouping` decorator takes one optional parameter:
|
||||
|
||||
- `flatten` determines how Flow elements are processed.
|
||||
If `False`, the callback function is called once per Flow element.
|
||||
If `True` (default), the callback is called for all Flow-Blocks of the Flow individually.
|
||||
The attribute `testB` can be attached to either the Flow or a Flow-Block regardless.
|
||||
|
||||
The `args` parameter of the `convert_simple_example()` function is a named tuple, it has three attributes:
|
||||
|
||||
1. The `record` points to the `Page` source which contains the tag.
|
||||
1. The `record` points to the `Page` record that contains the tag.
|
||||
2. The `key` tuple `(field-key, flow-index, flow-key)` tells which field is processed.
|
||||
For Flow types, `flow-index` and `flow-key` are set, otherwise they are `None`.
|
||||
3. The `field` value is the content of the processed field.
|
||||
The field value is reoughly equivalent to the following:
|
||||
The field value is equivalent to the following:
|
||||
|
||||
```python
|
||||
args.page[fieldKey].blocks[flowIndex].get(flowKey)
|
||||
k = args.key
|
||||
field = args.record[k.fieldKey].blocks[k.flowIndex].get(k.flowKey)
|
||||
```
|
||||
|
||||
The callback body **can** produce groupings but does not have to.
|
||||
If you choose to produce an entry, you have to `yield` a tuple pair of `(groupkey, extra-info)`.
|
||||
`groupkey` is used to combine & cluster pages and must be URL-safe.
|
||||
The `extra-info` is passed through to your template file.
|
||||
You can yield more than one entry per source or filter / ignore pages if you don't yield anything.
|
||||
|
||||
The template file can access and display the `extra-info`:
|
||||
Again, you can use all properties in your template.
|
||||
|
||||
```jinja2
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<b>Page: {{ child.title }}<b>
|
||||
<p>Custom field date: {{this.date}}</p>
|
||||
<ul>
|
||||
{%- for extra in extras %}
|
||||
<li>Name: {{ extra.val }}, Tag count: {{ extra.tags_in_page }}</li>
|
||||
{%- for child in this.children %}
|
||||
<li>page "{{child.path}}" with tags: {{child.tags}}</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
{%- endfor %}
|
||||
```
|
||||
|
||||
|
||||
@@ -154,11 +229,11 @@ The template file can access and display the `extra-info`:
|
||||
## Advanced example
|
||||
|
||||
Relevant files:
|
||||
```
|
||||
configs/advanced.ini
|
||||
packages/advanced-example/lektor_advanced.py
|
||||
templates/example-advanced.html
|
||||
```
|
||||
|
||||
- [`configs/advanced.ini`](./configs/advanced.ini)
|
||||
- [`packages/advanced-example/lektor_advanced.py`](./packages/advanced-example/lektor_advanced.py)
|
||||
- [`templates/example-advanced.html`](./templates/example-advanced.html)
|
||||
|
||||
|
||||
The following example is similar to the previous one.
|
||||
Except that it loads a config file and replaces in-text occurrences of `{{Tagname}}` with `<a href="/tag/">Tagname</a>`.
|
||||
@@ -166,44 +241,109 @@ Except that it loads a config file and replaces in-text occurrences of `{{Tagnam
|
||||
```python
|
||||
def on_groupby_before_build_all(self, groupby, builder, **extra):
|
||||
# load config
|
||||
regex = re.compile(self.get_config().get('match'))
|
||||
# since we load and use a config file, we need to track the dependency
|
||||
@groupby.depends_on(self.config_filename)
|
||||
@groupby.watch('/', 'testC', slug='advanced/{group}/',
|
||||
template='example-advanced.html')
|
||||
config = self.get_config()
|
||||
regex = config.get('testC.pattern.match')
|
||||
try:
|
||||
regex = re.compile(regex)
|
||||
except Exception as e:
|
||||
print('inlinetags.regex not valid: ' + str(e))
|
||||
return
|
||||
|
||||
# load config directly (which also tracks dependency)
|
||||
watcher = groupby.add_watcher('testC', config, pre_build=True)
|
||||
|
||||
@watcher.grouping()
|
||||
def convert_replace_example(args):
|
||||
# args.field assumed to be Markdown
|
||||
obj = args.field.source
|
||||
url_map = {} # type Dict[str, str]
|
||||
for match in regex.finditer(obj):
|
||||
tag = match.group(1)
|
||||
yield slugify(tag), tag
|
||||
vobj = yield tag
|
||||
if not hasattr(vobj, 'custom_attr'):
|
||||
vobj.custom_attr = []
|
||||
vobj.custom_attr.append(tag)
|
||||
url_map[tag] = vobj.url_path
|
||||
print('[advanced] slugify:', tag, '->', vobj.key)
|
||||
|
||||
def _fn(match: re.Match) -> str:
|
||||
tag = match.group(1)
|
||||
return f'<a href="/advanced/{slugify(tag)}/">{tag}</a>'
|
||||
return f'<a href="{url_map[tag]}">{tag}</a>'
|
||||
args.field.source = regex.sub(_fn, obj)
|
||||
```
|
||||
|
||||
One **important** thing to notice is, we use `on_groupby_before_build_all` to register our callback function.
|
||||
This is required because we would like to modify the source **before** it is written to disk.
|
||||
If you look back to the [simple example](#simple-example), we used `on_groupby_after_build_all` because we did not care when it is executed.
|
||||
Generally, it makes little difference which one you use (`on-after` is likely less busy).
|
||||
Just know that you can process the source before or after it is build.
|
||||
Notice, `add_watcher` accepts a config file as parameter which keeps also track of dependencies and rebuilds pages when you edit the config file.
|
||||
Further, the `yield` call returns a `GroupBySource` virtual object.
|
||||
You can use this object to add custom static attributes (similar to dynamic attributes with the `.fields` subsection config).
|
||||
|
||||
Not all attributes are available at this time, as the grouping is still in progress.
|
||||
But you can use `vobj.url_path` to get the target URL or `vobj.key` to get the slugified object-key (substitutions from `key_map` are already applied).
|
||||
|
||||
Usually, the grouping is postponed until the very end of the build process.
|
||||
However, in this case we want to modify the source before it is build by Lektor.
|
||||
For this situation we need to set `pre_build=True` in our `groupby.add_watcher()` call.
|
||||
All watcher with this flag will be processed before any Page is built.
|
||||
**Note:** If you can, avoid this performance regression.
|
||||
The grouping for these watchers will be performed each time you navigate from one page to another.
|
||||
|
||||
This example uses a Markdown model type as source.
|
||||
For Markdown fields, we can modify the `source` attribute directly.
|
||||
All other field typed need to be accessed via `args.record` key indirection.
|
||||
All other field types need to be accessed via `args.record` key indirection (see [simple example](#simple-example)).
|
||||
|
||||
```ini
|
||||
[testC]
|
||||
root = /
|
||||
slug = "advanced/{}/".format(this.key)
|
||||
template = example-advanced.html
|
||||
|
||||
[testC.pattern]
|
||||
match = {{([^}]{1,32})}}
|
||||
```
|
||||
|
||||
Lastly, the config file contains a regular expression which matches `{{` + any string less than 32 characters + `}}`.
|
||||
Notice, the parenthesis (`()`) will match the inner part but the replace function (`re.sub`) will remove the `{{}}` too.
|
||||
The config file takes the same parameters as the [config example](#quick-config).
|
||||
We introduced a new config option `testC.pattern.match`.
|
||||
This regular expression matches `{{` + any string less than 32 characters + `}}`.
|
||||
Notice, the parenthesis (`()`) will match only the inner part, thus the replace function (`re.sub`) removes the `{{}}`.
|
||||
|
||||
If the user changes the regex pattern in the config file, we need to rebuild all tags.
|
||||
For this purpose we need to track changes to the config file.
|
||||
This is done by calling:
|
||||
|
||||
```python
|
||||
@groupby.depends_on(file1, file2, ...)
|
||||
|
||||
## Misc
|
||||
|
||||
### Omit output with empty slugs
|
||||
|
||||
It was shortly mentioned above that slugs can be `None` (e.g., manually set to `slug = None`).
|
||||
This is useful if you do not want to create subpages but rather an index page containing all groups.
|
||||
You can combine this with the next use-case.
|
||||
|
||||
|
||||
### Index pages & Group query + filter
|
||||
|
||||
```jinja2
|
||||
{%- for x in this|vgroups(keys=['TestA', 'TestB'], fields=[], flows=[], recursive=True, order_by='key_obj') %}
|
||||
<a href="{{ x|url }}">({{ x.key_obj }})</a>
|
||||
{%- endfor %}
|
||||
```
|
||||
|
||||
You can query the groups of any parent node (including those without slug).
|
||||
[`templates/page.html`](./templates/page.html) uses this.
|
||||
The keys (`'TestA', 'TestB'`) can be omitted which will return all groups of all attributes (you can still filter them with `x.config.key == 'TestC'`).
|
||||
The `fields` and `flows` params are also optional.
|
||||
With these you can match groups in `args.key.fieldKey` and `args.key.flowKey`.
|
||||
For example, if you have a “main-tags” field and an “additional-tags” field and you want to show the main-tags in a preview but both tags on a detail page.
|
||||
|
||||
|
||||
### Sorting groups
|
||||
|
||||
Sorting is supported for the `vgroups` filter as well as for the children of each group (via config subsection `.children.order_by`).
|
||||
Coming back to the previous example, `order_by` can be either a comma-separated string of keys or a list of strings.
|
||||
Again, same [order-by](https://www.getlektor.com/docs/guides/page-order/) rules apply as for any other Lektor `Record`.
|
||||
Only this time, the attributes of the `GroupBy` object are used for sorting (including those you defined in the `.fields` subsection).
|
||||
|
||||
|
||||
### Pagination
|
||||
|
||||
You may use the `.pagination` subsection or `watcher.config.set_pagination()` to configure a pagination controller.
|
||||
The `url_path` of a paginated Page depends on your `slug` value.
|
||||
If the slug ends on `/` or `/index.html`, Lektor will append `page/2/index.html` to the second page.
|
||||
If the slug contains a `.` (e.g. `/a/{key}.html`), Lektor will insert `page2` in front of the extension (e.g., `/a/{key}page2.html`).
|
||||
If you supply a different `url_suffix`, for example “.X.”, those same two urls will become `.X./2/index.html` and `/a/{key}.X.2.html` respectively.
|
||||
|
||||
@@ -1 +1,15 @@
|
||||
[testC]
|
||||
root = /
|
||||
slug = "advanced/{}/".format(this.key)
|
||||
template = example-advanced.html
|
||||
|
||||
[testC.pattern]
|
||||
match = {{([^}]{1,32})}}
|
||||
|
||||
[testC.fields]
|
||||
desc = "Input object: {}, output key: {}".format(this.key_obj, this.key)
|
||||
|
||||
[testC.key_map]
|
||||
Blog = case-sensitive
|
||||
Two = three
|
||||
three = no-nested-replace
|
||||
|
||||
@@ -1,5 +1,22 @@
|
||||
[testA]
|
||||
enabled = True
|
||||
root = /
|
||||
slug = config/{group}.html
|
||||
slug = config/{key}.html
|
||||
template = example-config.html
|
||||
split = ' '
|
||||
key_obj_fn = '{}-z-{}'.format(X.upper(), ARGS.key.fieldKey) if X else None
|
||||
replace_none_key = unknown
|
||||
|
||||
[testA.children]
|
||||
order_by = -title, body
|
||||
|
||||
[testA.pagination]
|
||||
enabled = true
|
||||
per_page = 1
|
||||
url_suffix = .page.
|
||||
|
||||
[testA.fields]
|
||||
title = "Tagged: " ~ this.key_obj
|
||||
|
||||
[testA.key_map]
|
||||
Blog = News
|
||||
|
||||
@@ -1,31 +1,40 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from lektor.pluginsystem import Plugin
|
||||
from lektor.utils import slugify
|
||||
from typing import Generator
|
||||
import re
|
||||
from lektor_groupby import GroupBy, GroupByCallbackArgs
|
||||
|
||||
|
||||
class AdvancedGroupByPlugin(Plugin):
|
||||
def on_groupby_before_build_all(self, groupby, builder, **extra):
|
||||
def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
|
||||
# load config
|
||||
regex = self.get_config().get('match')
|
||||
config = self.get_config()
|
||||
regex = config.get('testC.pattern.match')
|
||||
try:
|
||||
regex = re.compile(regex)
|
||||
except Exception as e:
|
||||
print('inlinetags.regex not valid: ' + str(e))
|
||||
return
|
||||
|
||||
# since we load and use a config file, we need to track the dependency
|
||||
@groupby.depends_on(self.config_filename)
|
||||
@groupby.watch('/', 'testC', slug='advanced/{group}/',
|
||||
template='example-advanced.html')
|
||||
def convert_replace_example(args):
|
||||
# load config directly (which also tracks dependency)
|
||||
watcher = groupby.add_watcher('testC', config, pre_build=True)
|
||||
|
||||
@watcher.grouping()
|
||||
def _replace(args: GroupByCallbackArgs) -> Generator[str, str, None]:
|
||||
# args.field assumed to be Markdown
|
||||
obj = args.field.source
|
||||
url_map = {} # type Dict[str, str]
|
||||
for match in regex.finditer(obj):
|
||||
tag = match.group(1)
|
||||
yield slugify(tag), tag
|
||||
vobj = yield tag
|
||||
if not hasattr(vobj, 'custom_attr'):
|
||||
vobj.custom_attr = []
|
||||
# update static custom attribute
|
||||
vobj.custom_attr.append(tag)
|
||||
url_map[tag] = vobj.url_path
|
||||
print('[advanced] slugify:', tag, '->', vobj.key)
|
||||
|
||||
def _fn(match: re.Match) -> str:
|
||||
tag = match.group(1)
|
||||
return f'<a href="/advanced/{slugify(tag)}/">{tag}</a>'
|
||||
return f'<a href="{url_map[tag]}">{tag}</a>'
|
||||
args.field.source = regex.sub(_fn, obj)
|
||||
|
||||
@@ -1,17 +1,34 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from lektor.pluginsystem import Plugin
|
||||
from lektor.utils import slugify
|
||||
from typing import Iterator, Tuple
|
||||
from datetime import datetime
|
||||
from lektor_groupby import GroupBy, GroupByCallbackArgs
|
||||
|
||||
|
||||
class SimpleGroupByPlugin(Plugin):
|
||||
def on_groupby_after_build_all(self, groupby, builder, **extra):
|
||||
@groupby.watch('/blog', 'testB', slug='simple/{group}/index.html',
|
||||
template='example-simple.html', flatten=True)
|
||||
def convert_simple_example(args):
|
||||
def on_groupby_before_build_all(self, groupby: GroupBy, builder, **extra):
|
||||
watcher = groupby.add_watcher('testB', {
|
||||
'root': '/blog',
|
||||
'slug': 'simple/{key}/index.html',
|
||||
'template': 'example-simple.html',
|
||||
'key_obj_fn': 'X.upper() if X else "empty"',
|
||||
'replace_none_key': 'unknown',
|
||||
})
|
||||
watcher.config.set_key_map({'Foo': 'bar'})
|
||||
watcher.config.set_fields({'date': datetime.now()})
|
||||
watcher.config.set_order_by('-title,body')
|
||||
watcher.config.set_pagination(
|
||||
enabled=True,
|
||||
per_page=1,
|
||||
url_suffix='p',
|
||||
)
|
||||
|
||||
@watcher.grouping(flatten=True)
|
||||
def fn_simple(args: GroupByCallbackArgs) -> Iterator[Tuple[str, dict]]:
|
||||
# Yield groups
|
||||
value = args.field # list type since model is 'strings' type
|
||||
value = args.field # type: list # since model is 'strings' type
|
||||
for tag in value:
|
||||
yield slugify(tag), {'val': tag, 'tags_in_page': len(value)}
|
||||
yield tag
|
||||
# Everything below is just for documentation purposes
|
||||
page = args.record # extract additional info from source
|
||||
fieldKey, flowIndex, flowKey = args.key # or get field index
|
||||
@@ -19,6 +36,6 @@ class SimpleGroupByPlugin(Plugin):
|
||||
obj = page[fieldKey]
|
||||
else:
|
||||
obj = page[fieldKey].blocks[flowIndex].get(flowKey)
|
||||
print('page:', page)
|
||||
print(' obj:', obj)
|
||||
print()
|
||||
print('[simple] page:', page)
|
||||
print('[simple] obj:', obj)
|
||||
print('[simple] ')
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
<h2>Path: {{ this | url(absolute=True) }}</h2>
|
||||
<div>This is: {{this}}</div>
|
||||
<div>Children: {{this.children}}</div>
|
||||
<p>This is: {{this}}</p>
|
||||
<p>Custom field, desc: "{{this.desc}}"</p>
|
||||
<p>Custom static, seen objects: {{this.custom_attr}}</p>
|
||||
<p>Children: {{this.children.all()}}</p>
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
<h2>Path: {{ this | url(absolute=True) }}</h2>
|
||||
<div>This is: {{this}}</div>
|
||||
<p>This is: {{this}}</p>
|
||||
<p>Object: "{{this.key_obj}}", Key: "{{this.key}}"</p>
|
||||
<p>Custom field title: "{{this.title}}"</p>
|
||||
<ul>
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<li>Page: {{ child.path }}, Tags: {{ extras }}</li>
|
||||
{%- for child in this.children %}
|
||||
<li>Child: <a href="{{child|url}}">{{child.title}}</a> ({{child.path}})</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
@@ -1,12 +1,10 @@
|
||||
<h2>Path: {{ this | url(absolute=True) }}</h2>
|
||||
<div>This is: {{this}}</div>
|
||||
<p>This is: {{this}}</p>
|
||||
<p>Key: {{this.key}}</p>
|
||||
<p>Object: {{this.key_obj}}</p>
|
||||
<p>Custom field date: {{this.date}}</p>
|
||||
<ul>
|
||||
{%- for child, extras in this.children.items() %}
|
||||
<li>Page: {{ child.path }}</li>
|
||||
<ul>
|
||||
{%- for extra in extras %}
|
||||
Name: {{ extra.val }}, Tag count: {{ extra.tags_in_page }}
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
{%- for child in this.children %}
|
||||
<li>page "{{child.path}}" with tags: {{child.tags}}</li>
|
||||
{%- endfor %}
|
||||
</ul>
|
||||
@@ -18,20 +18,12 @@ main { margin: 3em; }
|
||||
{% block body %}{{ this.body }}{% endblock %}
|
||||
</main>
|
||||
<footer>
|
||||
<div>Simple Tags:
|
||||
{% for tag in ['blog','directory','blog-post','initial','samegroup'] %}
|
||||
<a href="/blog/simple/{{tag}}/">({{tag}})</a>
|
||||
{% endfor %}
|
||||
</div>
|
||||
<div>Config Tags:
|
||||
{% for tag in ['root','blog','directory','blog-post','initial','samegroup'] %}
|
||||
<a href="/config/{{tag}}.html">({{tag}})</a>
|
||||
{% endfor %}
|
||||
</div>
|
||||
<div>Advanced Tags:
|
||||
{% for tag in ['tag','two','blog'] %}
|
||||
<a href="/advanced/{{tag}}/">({{tag}})</a>
|
||||
{% endfor %}
|
||||
{%- for k, v in [('testA','Config'),('testB','Simple'),('testC','Advanced')] %}
|
||||
<div>{{v}} Tags:
|
||||
{%- for x in this|vgroups(k, recursive=True, order_by='key_obj') %}
|
||||
<a href="{{ x|url }}">({{x.key}})</a>
|
||||
{%- endfor %}
|
||||
</div>
|
||||
{%- endfor %}
|
||||
</footer>
|
||||
</body>
|
||||
|
||||
@@ -1,498 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
from lektor.db import Database, Record # typing
|
||||
from lektor.build_programs import BuildProgram
|
||||
from lektor.builder import Artifact, Builder # typing
|
||||
from lektor.pluginsystem import Plugin
|
||||
from lektor.reporter import reporter
|
||||
from lektor.sourceobj import SourceObject, VirtualSourceObject
|
||||
from lektor.types.flow import Flow, FlowType
|
||||
from lektor.utils import bool_from_string, build_url, prune_file_and_folder
|
||||
# for quick config
|
||||
from lektor.utils import slugify
|
||||
|
||||
from typing import Tuple, Dict, Set, List, NamedTuple
|
||||
from typing import NewType, Optional, Iterator, Callable, Iterable
|
||||
|
||||
VPATH = '@groupby' # potentially unsafe. All matching entries are pruned.
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Typing
|
||||
# -----------------------------------
|
||||
AttributeKey = NewType('AttributeKey', str) # attribute of lektor model
|
||||
GroupKey = NewType('GroupKey', str) # key of group-by
|
||||
|
||||
|
||||
class ResolverConf(NamedTuple):
|
||||
attrib: AttributeKey
|
||||
group: GroupKey
|
||||
slug: str
|
||||
|
||||
|
||||
class FieldKeyPath(NamedTuple):
|
||||
fieldKey: str
|
||||
flowIndex: Optional[int] = None
|
||||
flowKey: Optional[str] = None
|
||||
|
||||
|
||||
class GroupByCallbackArgs(NamedTuple):
|
||||
record: Record
|
||||
key: FieldKeyPath
|
||||
field: object # lektor model data-field value
|
||||
|
||||
|
||||
GroupingCallback = Callable[[GroupByCallbackArgs],
|
||||
Iterator[Tuple[GroupKey, object]]]
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# VirtualSource & BuildProgram
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupBySource(VirtualSourceObject):
|
||||
'''
|
||||
Holds information for a single group/cluster.
|
||||
This object is accessible in your template file.
|
||||
Attributes: record, attrib, group, slug, template, children
|
||||
|
||||
:DEFAULTS:
|
||||
slug: "{attrib}/{group}/index.html"
|
||||
template: "groupby-attribute.html"
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
record: Record,
|
||||
attrib: AttributeKey,
|
||||
group: GroupKey, *,
|
||||
slug: Optional[str] = None, # default: "{attrib}/{group}/index.html"
|
||||
template: Optional[str] = None # default: "groupby-attrib.html"
|
||||
) -> None:
|
||||
super().__init__(record)
|
||||
self.attrib = attrib
|
||||
self.group = group
|
||||
self.template = template or 'groupby-{}.html'.format(self.attrib)
|
||||
# custom user path
|
||||
slug = slug or '{attrib}/{group}/index.html'
|
||||
slug = slug.replace('{attrib}', self.attrib)
|
||||
slug = slug.replace('{group}', self.group)
|
||||
if slug.endswith('/index.html'):
|
||||
slug = slug[:-10]
|
||||
self.slug = slug
|
||||
# user adjustable after init
|
||||
self.children = {} # type: Dict[Record, List[object]]
|
||||
self.dependencies = set() # type: Set[str]
|
||||
|
||||
@property
|
||||
def path(self) -> str:
|
||||
# Used in VirtualSourceInfo, used to prune VirtualObjects
|
||||
return f'{self.record.path}{VPATH}/{self.attrib}/{self.group}'
|
||||
|
||||
@property
|
||||
def url_path(self) -> str:
|
||||
# Actual path to resource as seen by the browser
|
||||
return build_url([self.record.path, self.slug])
|
||||
|
||||
def iter_source_filenames(self) -> Iterator[str]:
|
||||
''' Enumerate all dependencies '''
|
||||
if self.dependencies:
|
||||
yield from self.dependencies
|
||||
for record in self.children:
|
||||
yield from record.iter_source_filenames()
|
||||
|
||||
def __str__(self) -> str:
|
||||
txt = '<GroupBySource'
|
||||
for x in ['attrib', 'group', 'slug', 'template']:
|
||||
txt += ' {}="{}"'.format(x, getattr(self, x))
|
||||
return txt + ' children={}>'.format(len(self.children))
|
||||
|
||||
|
||||
class GroupByBuildProgram(BuildProgram):
|
||||
''' Generate Build-Artifacts and write files. '''
|
||||
|
||||
def produce_artifacts(self) -> None:
|
||||
url = self.source.url_path
|
||||
if url.endswith('/'):
|
||||
url += 'index.html'
|
||||
self.declare_artifact(url, sources=list(
|
||||
self.source.iter_source_filenames()))
|
||||
GroupByPruner.track(url)
|
||||
|
||||
def build_artifact(self, artifact: Artifact) -> None:
|
||||
self.source.pad.db.track_record_dependency(self.source)
|
||||
artifact.render_template_into(self.source.template, this=self.source)
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Helper
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupByPruner:
|
||||
'''
|
||||
Static collector for build-artifact urls.
|
||||
All non-tracked VPATH-urls will be pruned after build.
|
||||
'''
|
||||
_cache: Set[str] = set()
|
||||
# Note: this var is static or otherwise two instances of
|
||||
# GroupByCreator would prune each others artifacts.
|
||||
|
||||
@classmethod
|
||||
def track(cls, url: str) -> None:
|
||||
''' Add url to build cache to prevent pruning. '''
|
||||
cls._cache.add(url.lstrip('/'))
|
||||
|
||||
@classmethod
|
||||
def prune(cls, builder: Builder) -> None:
|
||||
''' Remove previously generated, unreferenced Artifacts. '''
|
||||
dest_path = builder.destination_path
|
||||
con = builder.connect_to_database()
|
||||
try:
|
||||
with builder.new_build_state() as build_state:
|
||||
for url, file in build_state.iter_artifacts():
|
||||
if url.lstrip('/') in cls._cache:
|
||||
continue # generated in this build-run
|
||||
infos = build_state.get_artifact_dependency_infos(url, [])
|
||||
for v_path, _ in infos:
|
||||
if VPATH not in v_path:
|
||||
continue # we only care about groupby Virtuals
|
||||
reporter.report_pruned_artifact(url)
|
||||
prune_file_and_folder(file.filename, dest_path)
|
||||
build_state.remove_artifact(url)
|
||||
break # there is only one VPATH-entry per source
|
||||
finally:
|
||||
con.close()
|
||||
cls._cache.clear()
|
||||
|
||||
|
||||
class GroupByModelReader:
|
||||
''' Find models and flow-models which contain attrib '''
|
||||
|
||||
def __init__(self, db: Database, attrib: AttributeKey) -> None:
|
||||
self._flows = {} # type: Dict[str, Set[str]]
|
||||
self._models = {} # type: Dict[str, Dict[str, str]]
|
||||
# find flow blocks with attrib
|
||||
for key, flow in db.flowblocks.items():
|
||||
tmp1 = set(f.name for f in flow.fields
|
||||
if bool_from_string(f.options.get(attrib, False)))
|
||||
if tmp1:
|
||||
self._flows[key] = tmp1
|
||||
# find models with attrib or flow-blocks containing attrib
|
||||
for key, model in db.datamodels.items():
|
||||
tmp2 = {} # Dict[str, str]
|
||||
for field in model.fields:
|
||||
if bool_from_string(field.options.get(attrib, False)):
|
||||
tmp2[field.name] = '*' # include all children
|
||||
elif isinstance(field.type, FlowType) and self._flows:
|
||||
# only processed if at least one flow has attrib
|
||||
fbs = field.type.flow_blocks
|
||||
# if fbs == None, all flow-blocks are allowed
|
||||
if fbs is None or any(x in self._flows for x in fbs):
|
||||
tmp2[field.name] = '?' # only some flow blocks
|
||||
if tmp2:
|
||||
self._models[key] = tmp2
|
||||
|
||||
def read(
|
||||
self,
|
||||
record: Record,
|
||||
flatten: bool = False
|
||||
) -> Iterator[Tuple[FieldKeyPath, object]]:
|
||||
'''
|
||||
Enumerate all fields of a Record with attrib = True.
|
||||
Flows are either returned directly (flatten=False) or
|
||||
expanded so that each flow-block is yielded (flatten=True)
|
||||
'''
|
||||
assert isinstance(record, Record)
|
||||
for r_key, subs in self._models.get(record.datamodel.id, {}).items():
|
||||
if subs == '*': # either normal field or flow type (all blocks)
|
||||
field = record[r_key]
|
||||
if flatten and isinstance(field, Flow):
|
||||
for i, flow in enumerate(field.blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key, block in flow._data.items():
|
||||
if f_key.startswith('_'): # e.g., _flowblock
|
||||
continue
|
||||
yield FieldKeyPath(r_key, i, f_key), block
|
||||
else:
|
||||
yield FieldKeyPath(r_key), field
|
||||
else: # always flow type (only some blocks)
|
||||
for i, flow in enumerate(record[r_key].blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key in self._flows.get(flowtype, []):
|
||||
yield FieldKeyPath(r_key, i, f_key), flow[f_key]
|
||||
|
||||
|
||||
class GroupByState:
|
||||
''' Holds and updates a groupby build state. '''
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.state = {} # type: Dict[GroupKey, Dict[Record, List]]
|
||||
self._processed = set() # type: Set[Record]
|
||||
|
||||
def __contains__(self, record: Record) -> bool:
|
||||
''' Returns True if record was already processed. '''
|
||||
return record.path in self._processed
|
||||
|
||||
def items(self) -> Iterable[Tuple[GroupKey, Dict]]:
|
||||
''' Iterable with (GroupKey, {record: extras}) tuples. '''
|
||||
return self.state.items()
|
||||
|
||||
def add(self, record: Record, group: Dict[GroupKey, List]) -> None:
|
||||
''' Append groups if not processed already. '''
|
||||
if record.path not in self._processed:
|
||||
self._processed.add(record.path)
|
||||
for group_key, extras in group.items():
|
||||
if group_key in self.state:
|
||||
self.state[group_key][record] = extras
|
||||
else:
|
||||
self.state[group_key] = {record: extras}
|
||||
|
||||
|
||||
class GroupByWatcher:
|
||||
'''
|
||||
Callback is called with (Record, FieldKeyPath, field-value).
|
||||
Callback may yield one or more (group-key, extra-info) tuples.
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
root: str,
|
||||
attrib: AttributeKey,
|
||||
callback: GroupingCallback, *,
|
||||
slug: Optional[str] = None, # default: "{attrib}/{group}/index.html"
|
||||
template: Optional[str] = None # default: "groupby-attrib.html"
|
||||
) -> None:
|
||||
self.root = root
|
||||
self.attrib = attrib
|
||||
self.callback = callback
|
||||
self.slug = slug
|
||||
self.template = template
|
||||
# user editable attributes
|
||||
self.flatten = True # if False, dont explode FlowType
|
||||
self.dependencies = set() # type: Set[str]
|
||||
|
||||
def initialize(self, db: Database) -> None:
|
||||
''' Reset internal state. You must initialize before each build! '''
|
||||
self._state = GroupByState()
|
||||
self._model_reader = GroupByModelReader(db, self.attrib)
|
||||
|
||||
def should_process(self, node: SourceObject) -> bool:
|
||||
''' Check if record path is being watched. '''
|
||||
if isinstance(node, Record):
|
||||
p = node['_path'] # type: str
|
||||
return p.startswith(self.root) or p + '/' == self.root
|
||||
return False
|
||||
|
||||
def process(self, record: Record) -> None:
|
||||
'''
|
||||
Will iterate over all record fields and call the callback method.
|
||||
Each record is guaranteed to be processed only once.
|
||||
'''
|
||||
if record in self._state:
|
||||
return
|
||||
tmp = {}
|
||||
for key, field in self._model_reader.read(record, self.flatten):
|
||||
for ret in self.callback(GroupByCallbackArgs(record, key, field)):
|
||||
assert isinstance(ret, (tuple, list)), \
|
||||
'Must return tuple (group-key, extra-info)'
|
||||
group_key, extra = ret
|
||||
if group_key not in tmp:
|
||||
tmp[group_key] = [extra]
|
||||
else:
|
||||
tmp[group_key].append(extra)
|
||||
self._state.add(record, tmp)
|
||||
|
||||
def iter_sources(self, root: Record) -> Iterator[GroupBySource]:
|
||||
''' Prepare and yield GroupBySource elements. '''
|
||||
for group_key, children in self._state.items():
|
||||
src = GroupBySource(root, self.attrib, group_key,
|
||||
slug=self.slug, template=self.template)
|
||||
src.dependencies = self.dependencies
|
||||
src.children = children
|
||||
yield src
|
||||
|
||||
def __str__(self) -> str:
|
||||
txt = '<GroupByWatcher'
|
||||
for x in [
|
||||
'root', 'attrib', 'slug', 'template', 'flatten', 'dependencies'
|
||||
]:
|
||||
txt += ' {}="{}"'.format(x, getattr(self, x))
|
||||
return txt + '>'
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Main Component
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupByCreator:
|
||||
'''
|
||||
Process all children with matching conditions under specified page.
|
||||
Creates a grouping of pages with similar (self-defined) attributes.
|
||||
The grouping is performed only once per build.
|
||||
'''
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._watcher = [] # type: List[GroupByWatcher]
|
||||
self._results = {} # type: Dict[str, GroupBySource]
|
||||
self._resolve_map = {} # type: Dict[str, ResolverConf]
|
||||
|
||||
# ----------------
|
||||
# Add Observer
|
||||
# ----------------
|
||||
|
||||
def depends_on(self, *args: str) \
|
||||
-> Callable[[GroupByWatcher], GroupByWatcher]:
|
||||
''' Set GroupBySource dependency, e.g., a plugin config file. '''
|
||||
def _decorator(r: GroupByWatcher) -> GroupByWatcher:
|
||||
r.dependencies.update(list(args))
|
||||
return r
|
||||
return _decorator
|
||||
|
||||
def watch(
|
||||
self,
|
||||
root: str,
|
||||
attrib: AttributeKey, *,
|
||||
slug: Optional[str] = None, # default: "{attrib}/{group}/index.html"
|
||||
template: Optional[str] = None, # default: "groupby-attrib.html"
|
||||
flatten: bool = True, # if False, dont explode FlowType
|
||||
) -> Callable[[GroupingCallback], GroupByWatcher]:
|
||||
'''
|
||||
Decorator to subscribe to attrib-elements.
|
||||
(record, field-key, field) -> (group-key, extra-info)
|
||||
|
||||
:DEFAULTS:
|
||||
slug: "{attrib}/{group}/index.html"
|
||||
template: "groupby-attrib.html"
|
||||
'''
|
||||
root = root.rstrip('/') + '/'
|
||||
|
||||
def _decorator(fn: GroupingCallback) -> GroupByWatcher:
|
||||
w = GroupByWatcher(root, attrib, fn, slug=slug, template=template)
|
||||
w.flatten = flatten
|
||||
self._watcher.append(w)
|
||||
return w
|
||||
|
||||
return _decorator
|
||||
|
||||
# -----------
|
||||
# Builder
|
||||
# -----------
|
||||
|
||||
def clear_previous_results(self) -> None:
|
||||
''' Reset prvious results. Must be called before each build. '''
|
||||
self._watcher.clear()
|
||||
self._results.clear()
|
||||
self._resolve_map.clear()
|
||||
|
||||
def make_cluster(self, builder: Builder) -> None:
|
||||
''' Perform groupby, iterate over all children. '''
|
||||
if not self._watcher:
|
||||
return
|
||||
for w in self._watcher:
|
||||
w.initialize(builder.pad.db)
|
||||
|
||||
queue = builder.pad.get_all_roots() # type: List[SourceObject]
|
||||
while queue:
|
||||
record = queue.pop()
|
||||
self.queue_now(record)
|
||||
if hasattr(record, 'attachments'):
|
||||
queue.extend(record.attachments) # type: ignore[attr-defined]
|
||||
if hasattr(record, 'children'):
|
||||
queue.extend(record.children) # type: ignore[attr-defined]
|
||||
# build artifacts
|
||||
for w in self._watcher:
|
||||
root = builder.pad.get(w.root)
|
||||
for vobj in w.iter_sources(root):
|
||||
self._results[vobj.url_path] = vobj
|
||||
self._watcher.clear()
|
||||
|
||||
def queue_now(self, node: SourceObject) -> None:
|
||||
''' Process record immediatelly (No-Op if already processed). '''
|
||||
for w in self._watcher:
|
||||
if w.should_process(node): # ensures type Record
|
||||
w.process(node) # type: ignore[arg-type]
|
||||
|
||||
def build_all(self, builder: Builder) -> None:
|
||||
''' Create virtual objects and build sources. '''
|
||||
for url, x in sorted(self._results.items()):
|
||||
builder.build(x)
|
||||
self._resolve_map[url] = ResolverConf(x.attrib, x.group, x.slug)
|
||||
self._results.clear()
|
||||
|
||||
# -----------------
|
||||
# Path resolver
|
||||
# -----------------
|
||||
|
||||
def resolve_dev_server_path(
|
||||
self, node: SourceObject, pieces: List[str]
|
||||
) -> Optional[GroupBySource]:
|
||||
''' Dev server only: Resolves path/ -> path/index.html '''
|
||||
if not isinstance(node, Record):
|
||||
return None
|
||||
conf = self._resolve_map.get(build_url([node.url_path] + pieces))
|
||||
if not conf:
|
||||
return None
|
||||
return GroupBySource(node, conf.attrib, conf.group, slug=conf.slug)
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# Plugin Entry
|
||||
# -----------------------------------
|
||||
|
||||
|
||||
class GroupByPlugin(Plugin):
|
||||
name = 'GroupBy Plugin'
|
||||
description = 'Cluster arbitrary records with field attribute keyword.'
|
||||
|
||||
def on_setup_env(self, **extra: object) -> None:
|
||||
self.creator = GroupByCreator()
|
||||
self.env.add_build_program(GroupBySource, GroupByBuildProgram)
|
||||
|
||||
# resolve /tag/rss/ -> /tag/rss/index.html (local server only)
|
||||
@self.env.urlresolver
|
||||
def _(node: SourceObject, parts: List[str]) -> Optional[GroupBySource]:
|
||||
return self.creator.resolve_dev_server_path(node, parts)
|
||||
|
||||
def _load_quick_config(self) -> None:
|
||||
''' Load config file quick listeners. '''
|
||||
config = self.get_config()
|
||||
for attrib in config.sections():
|
||||
sect = config.section_as_dict(attrib)
|
||||
root = sect.get('root', '/')
|
||||
slug = sect.get('slug')
|
||||
temp = sect.get('template')
|
||||
split = sect.get('split')
|
||||
|
||||
@self.creator.depends_on(self.config_filename)
|
||||
@self.creator.watch(root, attrib, slug=slug, template=temp)
|
||||
def _fn(args: GroupByCallbackArgs) \
|
||||
-> Iterator[Tuple[GroupKey, object]]:
|
||||
val = args.field
|
||||
if isinstance(val, str):
|
||||
val = val.split(split) if split else [val] # make list
|
||||
if isinstance(val, list):
|
||||
for tag in val:
|
||||
yield slugify(tag), tag
|
||||
|
||||
def on_before_build_all(self, builder: Builder, **extra: object) -> None:
|
||||
self.creator.clear_previous_results()
|
||||
# let other plugins register their @groupby.watch functions
|
||||
self.emit('before-build-all', groupby=self.creator, builder=builder)
|
||||
self.creator.make_cluster(builder)
|
||||
|
||||
def on_before_build(self, source: SourceObject, **extra: object) -> None:
|
||||
# before-build may be called before before-build-all (issue #1017)
|
||||
# make sure it is evaluated immediatelly
|
||||
self.creator.queue_now(source)
|
||||
|
||||
def on_after_build_all(self, builder: Builder, **extra: object) -> None:
|
||||
self.emit('after-build-all', groupby=self.creator, builder=builder)
|
||||
self._load_quick_config()
|
||||
self.creator.make_cluster(builder)
|
||||
self.creator.build_all(builder)
|
||||
|
||||
def on_after_prune(self, builder: Builder, **extra: object) -> None:
|
||||
# TODO: find a better way to prune unreferenced elements
|
||||
GroupByPruner.prune(builder)
|
||||
4
lektor_groupby/__init__.py
Normal file
4
lektor_groupby/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
from .config import Config # noqa: F401
|
||||
from .groupby import GroupBy # noqa: F401
|
||||
from .plugin import GroupByPlugin # noqa: F401
|
||||
from .watcher import GroupByCallbackArgs # noqa: F401
|
||||
103
lektor_groupby/backref.py
Normal file
103
lektor_groupby/backref.py
Normal file
@@ -0,0 +1,103 @@
|
||||
from lektor.context import get_ctx
|
||||
from typing import TYPE_CHECKING, Set, Union, Iterable, Iterator
|
||||
import weakref
|
||||
from .util import split_strip
|
||||
if TYPE_CHECKING:
|
||||
from lektor.builder import Builder
|
||||
from lektor.db import Record
|
||||
from .groupby import GroupBy
|
||||
from .model import FieldKeyPath
|
||||
from .vobj import GroupBySource
|
||||
|
||||
|
||||
class WeakVGroupsList(list):
|
||||
def add(self, strong: 'FieldKeyPath', weak: 'GroupBySource') -> None:
|
||||
super().append((strong, weakref.ref(weak)))
|
||||
# super().append((strong, weak)) # strong-ref
|
||||
|
||||
|
||||
class GroupByRef:
|
||||
@staticmethod
|
||||
def of(builder: 'Builder') -> 'GroupBy':
|
||||
''' Get the GroupBy object of a builder. '''
|
||||
return builder.__groupby # type:ignore[attr-defined,no-any-return]
|
||||
|
||||
@staticmethod
|
||||
def set(builder: 'Builder', groupby: 'GroupBy') -> None:
|
||||
''' Set the GroupBy object of a builder. '''
|
||||
builder.__groupby = groupby # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class VGroups:
|
||||
@staticmethod
|
||||
def of(record: 'Record') -> WeakVGroupsList:
|
||||
'''
|
||||
Return the (weak) set of virtual objects of a page.
|
||||
Creates a new set if it does not exist yet.
|
||||
'''
|
||||
try:
|
||||
wset = record.__vgroups # type: ignore[attr-defined]
|
||||
except AttributeError:
|
||||
wset = WeakVGroupsList()
|
||||
record.__vgroups = wset # type: ignore[attr-defined]
|
||||
return wset # type: ignore[no-any-return]
|
||||
|
||||
@staticmethod
|
||||
def iter(
|
||||
record: 'Record',
|
||||
keys: Union[str, Iterable[str], None] = None,
|
||||
*,
|
||||
fields: Union[str, Iterable[str], None] = None,
|
||||
flows: Union[str, Iterable[str], None] = None,
|
||||
recursive: bool = False,
|
||||
order_by: Union[str, Iterable[str], None] = None,
|
||||
) -> Iterator['GroupBySource']:
|
||||
''' Extract all referencing groupby virtual objects from a page. '''
|
||||
# prepare filter
|
||||
if isinstance(keys, str):
|
||||
keys = [keys]
|
||||
if isinstance(fields, str):
|
||||
fields = [fields]
|
||||
if isinstance(flows, str):
|
||||
flows = [flows]
|
||||
# get GroupBy object
|
||||
ctx = get_ctx()
|
||||
if not ctx:
|
||||
raise NotImplementedError("Shouldn't happen, where is my context?")
|
||||
builder = ctx.build_state.builder
|
||||
GroupByRef.of(builder).make_once(keys) # ensure did cluster before use
|
||||
# find groups
|
||||
proc_list = [record]
|
||||
done_list = set() # type: Set[GroupBySource]
|
||||
while proc_list:
|
||||
page = proc_list.pop(0)
|
||||
if recursive and hasattr(page, 'children'):
|
||||
proc_list.extend(page.children)
|
||||
for key, vobj in VGroups.of(page):
|
||||
if fields and key.fieldKey not in fields:
|
||||
continue
|
||||
if flows and key.flowKey not in flows:
|
||||
continue
|
||||
if keys and vobj().config.key not in keys:
|
||||
continue
|
||||
done_list.add(vobj())
|
||||
|
||||
# manage config dependencies
|
||||
deps = set() # type: Set[str]
|
||||
for vobj in done_list:
|
||||
deps.update(vobj.config.dependencies)
|
||||
# ctx.record_virtual_dependency(vobj) # TODO: needed? works without
|
||||
for dep in deps:
|
||||
ctx.record_dependency(dep)
|
||||
|
||||
if order_by:
|
||||
if isinstance(order_by, str):
|
||||
order = split_strip(order_by, ',') # type: Iterable[str]
|
||||
elif isinstance(order_by, (list, tuple)):
|
||||
order = order_by
|
||||
else:
|
||||
raise AttributeError('order_by must be str or list type.')
|
||||
# using get_sort_key() of GroupBySource
|
||||
yield from sorted(done_list, key=lambda x: x.get_sort_key(order))
|
||||
else:
|
||||
yield from done_list
|
||||
200
lektor_groupby/config.py
Normal file
200
lektor_groupby/config.py
Normal file
@@ -0,0 +1,200 @@
|
||||
from inifile import IniFile
|
||||
from lektor.environment import Expression
|
||||
from lektor.context import Context
|
||||
from lektor.utils import slugify as _slugify
|
||||
from typing import (
|
||||
TYPE_CHECKING, Set, Dict, Optional, Union, Any, List, Generator
|
||||
)
|
||||
from .util import split_strip
|
||||
if TYPE_CHECKING:
|
||||
from lektor.sourceobj import SourceObject
|
||||
|
||||
|
||||
AnyConfig = Union['Config', IniFile, Dict]
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
''' Used to print a Lektor console error. '''
|
||||
|
||||
def __init__(
|
||||
self, key: str, field: str, expr: str, error: Union[Exception, str]
|
||||
):
|
||||
self.key = key
|
||||
self.field = field
|
||||
self.expr = expr
|
||||
self.error = error
|
||||
|
||||
def __str__(self) -> str:
|
||||
return 'Invalid config for [{}.{}] = "{}" – Error: {}'.format(
|
||||
self.key, self.field, self.expr, repr(self.error))
|
||||
|
||||
|
||||
class Config:
|
||||
'''
|
||||
Holds information for GroupByWatcher and GroupBySource.
|
||||
This object is accessible in your template file ({{this.config}}).
|
||||
|
||||
Available attributes:
|
||||
key, root, slug, template, enabled, dependencies, fields, key_map
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
key: str, *,
|
||||
root: Optional[str] = None, # default: "/"
|
||||
slug: Optional[str] = None, # default: "{attr}/{group}/index.html"
|
||||
template: Optional[str] = None, # default: "groupby-{attr}.html"
|
||||
replace_none_key: Optional[str] = None, # default: None
|
||||
key_obj_fn: Optional[str] = None, # default: None
|
||||
) -> None:
|
||||
self.key = key
|
||||
self.root = (root or '/').rstrip('/') or '/'
|
||||
self.slug = slug or (key + '/{key}/') # key = GroupBySource.key
|
||||
self.template = template or f'groupby-{self.key}.html'
|
||||
self.replace_none_key = replace_none_key
|
||||
self.key_obj_fn = key_obj_fn
|
||||
# editable after init
|
||||
self.enabled = True
|
||||
self.dependencies = set() # type: Set[str]
|
||||
self.fields = {} # type: Dict[str, Any]
|
||||
self.key_map = {} # type: Dict[str, str]
|
||||
self.pagination = {} # type: Dict[str, Any]
|
||||
self.order_by = None # type: Optional[List[str]]
|
||||
|
||||
def slugify(self, k: str) -> str:
|
||||
''' key_map replace and slugify. '''
|
||||
rv = self.key_map.get(k, k)
|
||||
return _slugify(rv) or rv # the `or` allows for example "_"
|
||||
|
||||
def set_fields(self, fields: Optional[Dict[str, Any]]) -> None:
|
||||
'''
|
||||
The fields dict is a mapping of attrib = Expression values.
|
||||
Each dict key will be added to the GroupBySource virtual object.
|
||||
Each dict value is passed through jinja context first.
|
||||
'''
|
||||
self.fields = fields or {}
|
||||
|
||||
def set_key_map(self, key_map: Optional[Dict[str, str]]) -> None:
|
||||
''' This mapping replaces group keys before slugify. '''
|
||||
self.key_map = key_map or {}
|
||||
|
||||
def set_pagination(
|
||||
self,
|
||||
enabled: Optional[bool] = None,
|
||||
per_page: Optional[int] = None,
|
||||
url_suffix: Optional[str] = None,
|
||||
items: Optional[str] = None,
|
||||
) -> None:
|
||||
''' Used for pagination. '''
|
||||
self.pagination = dict(
|
||||
enabled=enabled,
|
||||
per_page=per_page,
|
||||
url_suffix=url_suffix,
|
||||
items=items,
|
||||
)
|
||||
|
||||
def set_order_by(self, order_by: Optional[str]) -> None:
|
||||
''' If specified, children will be sorted according to keys. '''
|
||||
self.order_by = split_strip(order_by or '', ',') or None
|
||||
|
||||
def __repr__(self) -> str:
|
||||
txt = '<GroupByConfig'
|
||||
for x in ['enabled', 'key', 'root', 'slug', 'template', 'key_obj_fn']:
|
||||
txt += ' {}="{}"'.format(x, getattr(self, x))
|
||||
txt += f' fields="{", ".join(self.fields)}"'
|
||||
if self.order_by:
|
||||
txt += ' order_by="{}"'.format(' ,'.join(self.order_by))
|
||||
return txt + '>'
|
||||
|
||||
@staticmethod
|
||||
def from_dict(key: str, cfg: Dict[str, str]) -> 'Config':
|
||||
''' Set config fields manually. Allowed: key, root, slug, template. '''
|
||||
return Config(
|
||||
key=key,
|
||||
root=cfg.get('root'),
|
||||
slug=cfg.get('slug'),
|
||||
template=cfg.get('template'),
|
||||
replace_none_key=cfg.get('replace_none_key'),
|
||||
key_obj_fn=cfg.get('key_obj_fn'),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def from_ini(key: str, ini: IniFile) -> 'Config':
|
||||
''' Read and parse ini file. Also adds dependency tracking. '''
|
||||
cfg = ini.section_as_dict(key) # type: Dict[str, str]
|
||||
conf = Config.from_dict(key, cfg)
|
||||
conf.enabled = ini.get_bool(key + '.enabled', True)
|
||||
conf.dependencies.add(ini.filename)
|
||||
conf.set_fields(ini.section_as_dict(key + '.fields'))
|
||||
conf.set_key_map(ini.section_as_dict(key + '.key_map'))
|
||||
conf.set_pagination(
|
||||
enabled=ini.get_bool(key + '.pagination.enabled', None),
|
||||
per_page=ini.get_int(key + '.pagination.per_page', None),
|
||||
url_suffix=ini.get(key + '.pagination.url_suffix'),
|
||||
items=ini.get(key + '.pagination.items'),
|
||||
)
|
||||
conf.set_order_by(ini.get(key + '.children.order_by', None))
|
||||
return conf
|
||||
|
||||
@staticmethod
|
||||
def from_any(key: str, config: AnyConfig) -> 'Config':
|
||||
assert isinstance(config, (Config, IniFile, Dict))
|
||||
if isinstance(config, Config):
|
||||
return config
|
||||
elif isinstance(config, IniFile):
|
||||
return Config.from_ini(key, config)
|
||||
elif isinstance(config, Dict):
|
||||
return Config.from_dict(key, config)
|
||||
|
||||
# -----------------------------------
|
||||
# Field Expressions
|
||||
# -----------------------------------
|
||||
|
||||
def _make_expression(self, expr: Any, *, on: 'SourceObject', field: str) \
|
||||
-> Union[Expression, Any]:
|
||||
''' Create Expression and report any config error. '''
|
||||
if not isinstance(expr, str):
|
||||
return expr
|
||||
try:
|
||||
return Expression(on.pad.env, expr)
|
||||
except Exception as e:
|
||||
raise ConfigError(self.key, field, expr, e)
|
||||
|
||||
def eval_field(self, attr: str, *, on: 'SourceObject') \
|
||||
-> Union[Expression, Any]:
|
||||
''' Create an expression for a custom defined user field. '''
|
||||
# do not `gather_dependencies` because fields are evaluated on the fly
|
||||
# dependency tracking happens whenever a field is accessed
|
||||
return self._make_expression(
|
||||
self.fields[attr], on=on, field='fields.' + attr)
|
||||
|
||||
def eval_slug(self, key: str, *, on: 'SourceObject') -> Optional[str]:
|
||||
''' Either perform a "{key}" substitution or evaluate expression. '''
|
||||
cfg_slug = self.slug
|
||||
if not cfg_slug:
|
||||
return None
|
||||
if '{key}' in cfg_slug:
|
||||
if key:
|
||||
return cfg_slug.replace('{key}', key)
|
||||
else:
|
||||
raise ConfigError(self.key, 'slug', cfg_slug,
|
||||
'Cannot replace {key} with None')
|
||||
return None
|
||||
else:
|
||||
# TODO: do we need `gather_dependencies` here too?
|
||||
expr = self._make_expression(cfg_slug, on=on, field='slug')
|
||||
return expr.evaluate(on.pad, this=on, alt=on.alt) or None
|
||||
|
||||
def eval_key_obj_fn(self, *, on: 'SourceObject', context: Dict) -> Any:
|
||||
'''
|
||||
If `key_obj_fn` is set, evaluate field expression.
|
||||
Note: The function does not check whether `key_obj_fn` is set.
|
||||
Return: A Generator result is automatically unpacked into a list.
|
||||
'''
|
||||
exp = self._make_expression(self.key_obj_fn, on=on, field='key_obj_fn')
|
||||
with Context(pad=on.pad) as ctx:
|
||||
with ctx.gather_dependencies(self.dependencies.add):
|
||||
res = exp.evaluate(on.pad, this=on, alt=on.alt, values=context)
|
||||
if isinstance(res, Generator):
|
||||
res = list(res) # unpack for 1-to-n replacement
|
||||
return res
|
||||
118
lektor_groupby/groupby.py
Normal file
118
lektor_groupby/groupby.py
Normal file
@@ -0,0 +1,118 @@
|
||||
from lektor.builder import PathCache
|
||||
from lektor.db import Record # isinstance
|
||||
from lektor.reporter import reporter # build
|
||||
from typing import TYPE_CHECKING, List, Optional, Iterable
|
||||
from .config import Config
|
||||
from .watcher import Watcher
|
||||
if TYPE_CHECKING:
|
||||
from lektor.builder import Builder
|
||||
from lektor.sourceobj import SourceObject
|
||||
from .config import AnyConfig
|
||||
from .resolver import Resolver
|
||||
from .vobj import GroupBySource
|
||||
|
||||
|
||||
class GroupBy:
|
||||
'''
|
||||
Process all children with matching conditions under specified page.
|
||||
Creates a grouping of pages with similar (self-defined) attributes.
|
||||
The grouping is performed only once per build.
|
||||
'''
|
||||
|
||||
def __init__(self, resolver: 'Resolver') -> None:
|
||||
self._building = False
|
||||
self._watcher = [] # type: List[Watcher]
|
||||
self._results = [] # type: List[GroupBySource]
|
||||
self._pre_build_priority = [] # type: List[str] # config.key
|
||||
self.resolver = resolver
|
||||
|
||||
@property
|
||||
def isBuilding(self) -> bool:
|
||||
return self._building
|
||||
|
||||
def add_watcher(
|
||||
self, key: str, config: 'AnyConfig', *, pre_build: bool = False
|
||||
) -> Watcher:
|
||||
''' Init Config and add to watch list. '''
|
||||
w = Watcher(Config.from_any(key, config))
|
||||
self._watcher.append(w)
|
||||
if pre_build:
|
||||
self._pre_build_priority.append(w.config.key)
|
||||
return w
|
||||
|
||||
def queue_all(self, builder: 'Builder') -> None:
|
||||
''' Iterate full site-tree and queue all children. '''
|
||||
# remove disabled watchers
|
||||
self._watcher = [w for w in self._watcher if w.config.enabled]
|
||||
if not self._watcher:
|
||||
return
|
||||
# initialize remaining (enabled) watchers
|
||||
for w in self._watcher:
|
||||
w.initialize(builder.pad)
|
||||
# iterate over whole build tree
|
||||
queue = builder.pad.get_all_roots() # type: List[SourceObject]
|
||||
while queue:
|
||||
record = queue.pop()
|
||||
if hasattr(record, 'attachments'):
|
||||
queue.extend(record.attachments)
|
||||
if hasattr(record, 'children'):
|
||||
queue.extend(record.children)
|
||||
if isinstance(record, Record):
|
||||
for w in self._watcher:
|
||||
if w.should_process(record):
|
||||
w.remember(record)
|
||||
# build sources which need building before actual lektor build
|
||||
if self._pre_build_priority:
|
||||
self.make_once(self._pre_build_priority)
|
||||
self._pre_build_priority.clear()
|
||||
|
||||
def make_once(self, filter_keys: Optional[Iterable[str]] = None) -> None:
|
||||
'''
|
||||
Perform groupby, iter over sources with watcher callback.
|
||||
If `filter_keys` is set, ignore all other watchers.
|
||||
'''
|
||||
if not self._watcher:
|
||||
return
|
||||
remaining = []
|
||||
for w in self._watcher:
|
||||
# only process vobjs that are used somewhere
|
||||
if filter_keys and w.config.key not in filter_keys:
|
||||
remaining.append(w)
|
||||
continue
|
||||
self.resolver.reset(w.config.key)
|
||||
# these are used in the current context (or on `build_all`)
|
||||
for vobj in w.iter_sources():
|
||||
# add original source
|
||||
self._results.append(vobj)
|
||||
self.resolver.add(vobj)
|
||||
# and also add pagination sources
|
||||
for sub_vobj in vobj.__iter_pagination_sources__():
|
||||
self._results.append(sub_vobj)
|
||||
self.resolver.add(sub_vobj)
|
||||
# TODO: if this should ever run concurrently, pop() from watchers
|
||||
self._watcher = remaining
|
||||
|
||||
def build_all(
|
||||
self,
|
||||
builder: 'Builder',
|
||||
specific: Optional['GroupBySource'] = None
|
||||
) -> None:
|
||||
'''
|
||||
Build actual artifacts (if needed).
|
||||
If `specific` is set, only build the artifacts for that single vobj
|
||||
'''
|
||||
if not self._watcher and not self._results:
|
||||
return
|
||||
with reporter.build('groupby', builder): # type:ignore
|
||||
# in case no page used the |vgroups filter
|
||||
self.make_once([specific.config.key] if specific else None)
|
||||
self._building = True
|
||||
path_cache = PathCache(builder.env)
|
||||
for vobj in self._results:
|
||||
if specific and vobj.path != specific.path:
|
||||
continue
|
||||
if vobj.slug:
|
||||
builder.build(vobj, path_cache)
|
||||
del path_cache
|
||||
self._building = False
|
||||
self._results.clear() # garbage collect weak refs
|
||||
67
lektor_groupby/model.py
Normal file
67
lektor_groupby/model.py
Normal file
@@ -0,0 +1,67 @@
|
||||
from lektor.db import Database, Record # typing
|
||||
from lektor.types.flow import Flow, FlowType
|
||||
from lektor.utils import bool_from_string
|
||||
from typing import Set, Dict, Tuple, Any, NamedTuple, Optional, Iterator
|
||||
|
||||
|
||||
class FieldKeyPath(NamedTuple):
|
||||
fieldKey: str
|
||||
flowIndex: Optional[int] = None
|
||||
flowKey: Optional[str] = None
|
||||
|
||||
|
||||
class ModelReader:
|
||||
'''
|
||||
Find models and flow-models which contain attribute.
|
||||
Flows are either returned directly (flatten=False) or
|
||||
expanded so that each flow-block is yielded (flatten=True)
|
||||
'''
|
||||
|
||||
def __init__(self, db: Database, attr: str, flatten: bool = False) -> None:
|
||||
self.flatten = flatten
|
||||
self._flows = {} # type: Dict[str, Set[str]]
|
||||
self._models = {} # type: Dict[str, Dict[str, str]]
|
||||
# find flow blocks containing attribute
|
||||
for key, flow in db.flowblocks.items():
|
||||
tmp1 = set(f.name for f in flow.fields
|
||||
if bool_from_string(f.options.get(attr, False)))
|
||||
if tmp1:
|
||||
self._flows[key] = tmp1
|
||||
# find models and flow-blocks containing attribute
|
||||
for key, model in db.datamodels.items():
|
||||
tmp2 = {} # Dict[str, str]
|
||||
for field in model.fields:
|
||||
if bool_from_string(field.options.get(attr, False)):
|
||||
tmp2[field.name] = '*' # include all children
|
||||
elif isinstance(field.type, FlowType) and self._flows:
|
||||
# only processed if at least one flow has attr
|
||||
fbs = field.type.flow_blocks
|
||||
# if fbs == None, all flow-blocks are allowed
|
||||
if fbs is None or any(x in self._flows for x in fbs):
|
||||
tmp2[field.name] = '?' # only some flow blocks
|
||||
if tmp2:
|
||||
self._models[key] = tmp2
|
||||
|
||||
def read(self, record: Record) -> Iterator[Tuple[FieldKeyPath, Any]]:
|
||||
''' Enumerate all fields of a Record with attrib = True. '''
|
||||
assert isinstance(record, Record)
|
||||
for r_key, subs in self._models.get(record.datamodel.id, {}).items():
|
||||
field = record[r_key]
|
||||
if not field:
|
||||
yield FieldKeyPath(r_key), field
|
||||
continue
|
||||
if subs == '*': # either normal field or flow type (all blocks)
|
||||
if self.flatten and isinstance(field, Flow):
|
||||
for i, flow in enumerate(field.blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key, block in flow._data.items():
|
||||
if f_key.startswith('_'): # e.g., _flowblock
|
||||
continue
|
||||
yield FieldKeyPath(r_key, i, f_key), block
|
||||
else:
|
||||
yield FieldKeyPath(r_key), field
|
||||
else: # always flow type (only some blocks)
|
||||
for i, flow in enumerate(field.blocks):
|
||||
flowtype = flow['_flowblock']
|
||||
for f_key in self._flows.get(flowtype, []):
|
||||
yield FieldKeyPath(r_key, i, f_key), flow[f_key]
|
||||
29
lektor_groupby/pagination.py
Normal file
29
lektor_groupby/pagination.py
Normal file
@@ -0,0 +1,29 @@
|
||||
from lektor import datamodel
|
||||
from typing import TYPE_CHECKING, Any, Dict
|
||||
if TYPE_CHECKING:
|
||||
from lektor.environment import Environment
|
||||
from lektor.pagination import Pagination
|
||||
from lektor.sourceobj import SourceObject
|
||||
|
||||
|
||||
class PaginationConfig(datamodel.PaginationConfig):
|
||||
# because original method does not work for virtual sources.
|
||||
def __init__(self, env: 'Environment', config: Dict[str, Any], total: int):
|
||||
super().__init__(env, **config)
|
||||
self._total_items_count = total
|
||||
|
||||
@staticmethod
|
||||
def get_record_for_page(record: 'SourceObject', page_num: int) -> Any:
|
||||
for_page = getattr(record, '__for_page__', None)
|
||||
if callable(for_page):
|
||||
return for_page(page_num)
|
||||
return datamodel.PaginationConfig.get_record_for_page(record, page_num)
|
||||
|
||||
def count_total_items(self, record: 'SourceObject') -> int:
|
||||
''' Override super() to prevent a record.children query. '''
|
||||
return self._total_items_count
|
||||
|
||||
if TYPE_CHECKING:
|
||||
def get_pagination_controller(self, record: 'SourceObject') \
|
||||
-> 'Pagination':
|
||||
...
|
||||
102
lektor_groupby/plugin.py
Normal file
102
lektor_groupby/plugin.py
Normal file
@@ -0,0 +1,102 @@
|
||||
from lektor.assets import Asset # isinstance
|
||||
from lektor.db import Record # isinstance
|
||||
from lektor.pluginsystem import Plugin # subclass
|
||||
from typing import TYPE_CHECKING, Set, Iterator, Any
|
||||
from .backref import GroupByRef, VGroups
|
||||
from .groupby import GroupBy
|
||||
from .pruner import prune
|
||||
from .resolver import Resolver
|
||||
from .vobj import GroupBySource, GroupByBuildProgram
|
||||
if TYPE_CHECKING:
|
||||
from lektor.builder import Builder, BuildState
|
||||
from lektor.sourceobj import SourceObject
|
||||
from .watcher import GroupByCallbackArgs
|
||||
|
||||
|
||||
class GroupByPlugin(Plugin):
|
||||
name = 'GroupBy Plugin'
|
||||
description = 'Cluster arbitrary records with field attribute keyword.'
|
||||
|
||||
def on_setup_env(self, **extra: Any) -> None:
|
||||
self.resolver = Resolver(self.env)
|
||||
self.env.add_build_program(GroupBySource, GroupByBuildProgram)
|
||||
self.env.jinja_env.filters.update(vgroups=VGroups.iter)
|
||||
# kep track of already rebuilt GroupBySource artifacts
|
||||
self._is_build_all = False
|
||||
self._has_been_built = set() # type: Set[str]
|
||||
|
||||
def on_before_build_all(self, **extra: Any) -> None:
|
||||
self._is_build_all = True
|
||||
|
||||
def on_before_build(
|
||||
self, builder: 'Builder', source: 'SourceObject', **extra: Any
|
||||
) -> None:
|
||||
# before-build may be called before before-build-all (issue #1017)
|
||||
if isinstance(source, Asset):
|
||||
return
|
||||
# make GroupBySource available before building any Record artifact
|
||||
groupby = self._init_once(builder)
|
||||
# special handling for self-building of GroupBySource artifacts
|
||||
if isinstance(source, GroupBySource):
|
||||
if groupby.isBuilding: # build is during groupby.build_all()
|
||||
self._has_been_built.add(source.path)
|
||||
elif source.path not in self._has_been_built:
|
||||
groupby.build_all(builder, source) # needs rebuilding
|
||||
|
||||
def on_after_build(
|
||||
self, source: 'SourceObject', build_state: 'BuildState', **extra: Any
|
||||
) -> None:
|
||||
# a normal page update. We may need to re-build our GroupBySource
|
||||
if not self._is_build_all and isinstance(source, Record):
|
||||
if build_state.updated_artifacts:
|
||||
# TODO: instead of clear(), only remove affected GroupBySource
|
||||
# ideally, identify which file has triggered the re-build
|
||||
self._has_been_built.clear()
|
||||
|
||||
def on_after_build_all(self, builder: 'Builder', **extra: Any) -> None:
|
||||
# by now, most likely already built. So, build_all() is a no-op
|
||||
self._init_once(builder).build_all(builder)
|
||||
self._is_build_all = False
|
||||
|
||||
def on_after_prune(self, builder: 'Builder', **extra: Any) -> None:
|
||||
# TODO: find a better way to prune unreferenced elements
|
||||
prune(builder, self.resolver.files)
|
||||
|
||||
# ------------
|
||||
# internal
|
||||
# ------------
|
||||
|
||||
def _init_once(self, builder: 'Builder') -> GroupBy:
|
||||
try:
|
||||
return GroupByRef.of(builder)
|
||||
except AttributeError:
|
||||
groupby = GroupBy(self.resolver)
|
||||
GroupByRef.set(builder, groupby)
|
||||
|
||||
self._load_quick_config(groupby)
|
||||
# let other plugins register their @groupby.watch functions
|
||||
self.emit('before-build-all', groupby=groupby, builder=builder)
|
||||
groupby.queue_all(builder)
|
||||
return groupby
|
||||
|
||||
def _load_quick_config(self, groupby: GroupBy) -> None:
|
||||
''' Load config file quick listeners. '''
|
||||
config = self.get_config()
|
||||
for key in config.sections():
|
||||
if '.' in key: # e.g., key.fields and key.key_map
|
||||
continue
|
||||
|
||||
watcher = groupby.add_watcher(key, config)
|
||||
split = config.get(key + '.split') # type: str
|
||||
|
||||
@watcher.grouping()
|
||||
def _fn(args: 'GroupByCallbackArgs') -> Iterator[str]:
|
||||
val = args.field
|
||||
if isinstance(val, str) and val != '':
|
||||
val = map(str.strip, val.split(split)) if split else [val]
|
||||
elif isinstance(val, (bool, int, float)):
|
||||
val = [val]
|
||||
elif not val: # after checking for '', False, 0, and 0.0
|
||||
val = [None]
|
||||
if isinstance(val, (list, map)):
|
||||
yield from val
|
||||
77
lektor_groupby/pruner.py
Normal file
77
lektor_groupby/pruner.py
Normal file
@@ -0,0 +1,77 @@
|
||||
'''
|
||||
Usage:
|
||||
VirtualSourceObject.produce_artifacts()
|
||||
-> remember url and later supply as `current_urls`
|
||||
VirtualSourceObject.build_artifact()
|
||||
-> `get_ctx().record_virtual_dependency(VirtualPruner())`
|
||||
'''
|
||||
from lektor.reporter import reporter # report_pruned_artifact
|
||||
from lektor.sourceobj import VirtualSourceObject # subclass
|
||||
from lektor.utils import prune_file_and_folder
|
||||
import os
|
||||
from typing import TYPE_CHECKING, Set, List, Iterable
|
||||
if TYPE_CHECKING:
|
||||
from lektor.builder import Builder
|
||||
from sqlite3 import Connection
|
||||
|
||||
|
||||
class VirtualPruner(VirtualSourceObject):
|
||||
''' Indicate that a generated VirtualSourceObject has pruning support. '''
|
||||
VPATH = '/@VirtualPruner'
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._path = VirtualPruner.VPATH # if needed, add suffix variable
|
||||
|
||||
@property
|
||||
def path(self) -> str: # type: ignore[override]
|
||||
return self._path
|
||||
|
||||
|
||||
def prune(builder: 'Builder', current_urls: Iterable[str]) -> None:
|
||||
''' Removes previously generated, but now unreferenced Artifacts. '''
|
||||
dest_dir = builder.destination_path
|
||||
con = builder.connect_to_database()
|
||||
try:
|
||||
previous = _query_prunable(con)
|
||||
current = _normalize_urls(current_urls)
|
||||
to_be_pruned = previous.difference(current)
|
||||
for file in to_be_pruned:
|
||||
reporter.report_pruned_artifact(file) # type: ignore
|
||||
prune_file_and_folder(os.path.join(
|
||||
dest_dir, file.strip('/').replace('/', os.path.sep)), dest_dir)
|
||||
# if no exception raised, update db to remove obsolete references
|
||||
_prune_db_artifacts(con, list(to_be_pruned))
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
|
||||
# ---------------------------
|
||||
# Internal helper methods
|
||||
# ---------------------------
|
||||
|
||||
def _normalize_urls(urls: Iterable[str]) -> Set[str]:
|
||||
cache = set()
|
||||
for url in urls:
|
||||
if url.endswith('/'):
|
||||
url += 'index.html'
|
||||
cache.add(url.lstrip('/'))
|
||||
return cache
|
||||
|
||||
|
||||
def _query_prunable(conn: 'Connection') -> Set[str]:
|
||||
''' Query database for artifacts that have the VirtualPruner dependency '''
|
||||
cur = conn.cursor()
|
||||
cur.execute('SELECT artifact FROM artifacts WHERE source = ?',
|
||||
[VirtualPruner.VPATH])
|
||||
return set(x for x, in cur.fetchall())
|
||||
|
||||
|
||||
def _prune_db_artifacts(conn: 'Connection', urls: List[str]) -> None:
|
||||
''' Remove obsolete artifact references from database. '''
|
||||
MAX_VARS = 999 # Default SQLITE_MAX_VARIABLE_NUMBER.
|
||||
cur = conn.cursor()
|
||||
for i in range(0, len(urls), MAX_VARS):
|
||||
batch = urls[i: i + MAX_VARS]
|
||||
cur.execute('DELETE FROM artifacts WHERE artifact in ({})'.format(
|
||||
','.join(['?'] * len(batch))), batch)
|
||||
conn.commit()
|
||||
80
lektor_groupby/query.py
Normal file
80
lektor_groupby/query.py
Normal file
@@ -0,0 +1,80 @@
|
||||
# adapting https://github.com/dairiki/lektorlib/blob/master/lektorlib/query.py
|
||||
from lektor.constants import PRIMARY_ALT
|
||||
from lektor.db import Query # subclass
|
||||
from typing import TYPE_CHECKING, List, Optional, Generator, Iterable
|
||||
if TYPE_CHECKING:
|
||||
from lektor.db import Record, Pad
|
||||
|
||||
|
||||
class FixedRecordsQuery(Query):
|
||||
def __init__(
|
||||
self, pad: 'Pad', child_paths: Iterable[str], alt: str = PRIMARY_ALT
|
||||
):
|
||||
''' Query with a pre-defined list of children of type Record. '''
|
||||
super().__init__('/', pad, alt=alt)
|
||||
self.__child_paths = [x.lstrip('/') for x in child_paths]
|
||||
|
||||
def _get(
|
||||
self, path: str, persist: bool = True, page_num: Optional[int] = None
|
||||
) -> Optional['Record']:
|
||||
''' Internal getter for a single Record. '''
|
||||
if path not in self.__child_paths:
|
||||
return None
|
||||
if page_num is None:
|
||||
page_num = self._page_num
|
||||
return self.pad.get( # type: ignore[no-any-return]
|
||||
path, alt=self.alt, page_num=page_num, persist=persist)
|
||||
|
||||
def _iterate(self) -> Generator['Record', None, None]:
|
||||
''' Iterate over internal set of Record elements. '''
|
||||
# ignore self record dependency from super()
|
||||
for path in self.__child_paths:
|
||||
record = self._get(path, persist=False)
|
||||
if record is None:
|
||||
if self._page_num is not None:
|
||||
# Sanity check: ensure the unpaginated version exists
|
||||
unpaginated = self._get(path, persist=False, page_num=None)
|
||||
if unpaginated is not None:
|
||||
# Requested explicit page_num, but source does not
|
||||
# support pagination. Punt and skip it.
|
||||
continue
|
||||
raise RuntimeError('could not load source for ' + path)
|
||||
|
||||
is_attachment = getattr(record, 'is_attachment', False)
|
||||
if self._include_attachments and not is_attachment \
|
||||
or self._include_pages and is_attachment:
|
||||
continue
|
||||
if self._matches(record):
|
||||
yield record
|
||||
|
||||
def get_order_by(self) -> Optional[List[str]]:
|
||||
''' Return list of attribute strings for sort order. '''
|
||||
# ignore datamodel ordering from super()
|
||||
return self._order_by # type: ignore[no-any-return]
|
||||
|
||||
def count(self) -> int:
|
||||
''' Count matched objects. '''
|
||||
if self._pristine:
|
||||
return len(self.__child_paths)
|
||||
return super().count() # type: ignore[no-any-return]
|
||||
|
||||
@property
|
||||
def total(self) -> int:
|
||||
''' Return total entries count (without any filter). '''
|
||||
return len(self.__child_paths)
|
||||
|
||||
def get(self, path: str, page_num: Optional[int] = None) \
|
||||
-> Optional['Record']:
|
||||
''' Return Record with given path '''
|
||||
if path in self.__child_paths:
|
||||
return self._get(path, page_num=page_num)
|
||||
return None
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
if self._pristine:
|
||||
return len(self.__child_paths) > 0
|
||||
return super().__bool__()
|
||||
|
||||
if TYPE_CHECKING:
|
||||
def request_page(self, page_num: Optional[int]) -> 'FixedRecordsQuery':
|
||||
...
|
||||
95
lektor_groupby/resolver.py
Normal file
95
lektor_groupby/resolver.py
Normal file
@@ -0,0 +1,95 @@
|
||||
from lektor.db import Page # isinstance
|
||||
from typing import TYPE_CHECKING, NamedTuple, Dict, List, Set, Any, Optional
|
||||
from .util import build_url
|
||||
from .vobj import VPATH, GroupBySource
|
||||
if TYPE_CHECKING:
|
||||
from lektor.environment import Environment
|
||||
from lektor.sourceobj import SourceObject
|
||||
from .config import Config
|
||||
|
||||
|
||||
class ResolverEntry(NamedTuple):
|
||||
key: str
|
||||
key_obj: Any
|
||||
config: 'Config'
|
||||
page: Optional[int]
|
||||
|
||||
def equals(
|
||||
self, path: str, conf_key: str, vobj_key: str, page: Optional[int]
|
||||
) -> bool:
|
||||
return self.key == vobj_key \
|
||||
and self.config.key == conf_key \
|
||||
and self.config.root == path \
|
||||
and self.page == page
|
||||
|
||||
|
||||
class Resolver:
|
||||
'''
|
||||
Resolve virtual paths and urls ending in /.
|
||||
Init will subscribe to @urlresolver and @virtualpathresolver.
|
||||
'''
|
||||
|
||||
def __init__(self, env: 'Environment') -> None:
|
||||
self._data = {} # type: Dict[str, Dict[str, ResolverEntry]]
|
||||
env.urlresolver(self.resolve_server_path)
|
||||
env.virtualpathresolver(VPATH.lstrip('@'))(self.resolve_virtual_path)
|
||||
|
||||
@property
|
||||
def has_any(self) -> bool:
|
||||
return any(bool(x) for x in self._data.values())
|
||||
|
||||
@property
|
||||
def files(self) -> Set[str]:
|
||||
return set(y for x in self._data.values() for y in x.keys())
|
||||
|
||||
def reset(self, key: Optional[str] = None) -> None:
|
||||
''' Clear previously recorded virtual objects. '''
|
||||
if key:
|
||||
if key in self._data: # only delete if exists
|
||||
del self._data[key]
|
||||
else:
|
||||
self._data.clear()
|
||||
|
||||
def add(self, vobj: GroupBySource) -> None:
|
||||
''' Track new virtual object (only if slug is set). '''
|
||||
if vobj.slug:
|
||||
# `page_num = 1` overwrites `page_num = None` -> same url_path()
|
||||
if vobj.config.key not in self._data:
|
||||
self._data[vobj.config.key] = {}
|
||||
self._data[vobj.config.key][vobj.url_path] = ResolverEntry(
|
||||
vobj.key, vobj.key_obj, vobj.config, vobj.page_num)
|
||||
|
||||
# ------------
|
||||
# Resolver
|
||||
# ------------
|
||||
|
||||
def resolve_server_path(self, node: 'SourceObject', pieces: List[str]) \
|
||||
-> Optional[GroupBySource]:
|
||||
''' Local server only: resolve /tag/rss/ -> /tag/rss/index.html '''
|
||||
if isinstance(node, Page):
|
||||
url = build_url([node.url_path] + pieces)
|
||||
for subset in self._data.values():
|
||||
rv = subset.get(url)
|
||||
if rv:
|
||||
return GroupBySource(
|
||||
node, rv.key, rv.config, rv.page).finalize(rv.key_obj)
|
||||
return None
|
||||
|
||||
def resolve_virtual_path(self, node: 'SourceObject', pieces: List[str]) \
|
||||
-> Optional[GroupBySource]:
|
||||
''' Admin UI only: Prevent server error and null-redirect. '''
|
||||
# format: /path/to/page@groupby/{config-key}/{vobj-key}/{page-num}
|
||||
if isinstance(node, Page) and len(pieces) >= 2:
|
||||
path = node['_path'] # type: str
|
||||
conf_key, vobj_key, *optional_page = pieces
|
||||
page = None
|
||||
if optional_page:
|
||||
try:
|
||||
page = int(optional_page[0])
|
||||
except ValueError:
|
||||
pass
|
||||
for rv in self._data.get(conf_key, {}).values():
|
||||
if rv.equals(path, conf_key, vobj_key, page):
|
||||
return GroupBySource(
|
||||
node, rv.key, rv.config, rv.page).finalize(rv.key_obj)
|
||||
return None
|
||||
62
lektor_groupby/util.py
Normal file
62
lektor_groupby/util.py
Normal file
@@ -0,0 +1,62 @@
|
||||
from typing import List, Dict, Optional, TypeVar
|
||||
from typing import Callable, Any, Union, Generic
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
|
||||
def most_used_key(keys: List[T]) -> Optional[T]:
|
||||
''' Find string with most occurrences. '''
|
||||
if len(keys) < 3:
|
||||
return keys[0] if keys else None # TODO: first vs last occurrence
|
||||
best_count = 0
|
||||
best_key = None
|
||||
tmp = {} # type: Dict[T, int]
|
||||
for k in keys:
|
||||
num = (tmp[k] + 1) if k in tmp else 1
|
||||
tmp[k] = num
|
||||
if num > best_count: # TODO: (>) vs (>=), first vs last occurrence
|
||||
best_count = num
|
||||
best_key = k
|
||||
return best_key
|
||||
|
||||
|
||||
def split_strip(data: str, delimiter: str = ',') -> List[str]:
|
||||
''' Split by delimiter and strip each str separately. Omit if empty. '''
|
||||
ret = []
|
||||
for x in data.split(delimiter):
|
||||
x = x.strip()
|
||||
if x:
|
||||
ret.append(x)
|
||||
return ret
|
||||
|
||||
|
||||
def insert_before_ext(data: str, ins: str, delimiter: str = '.') -> str:
|
||||
''' Insert text before last index of delimeter (or at the end). '''
|
||||
assert delimiter in data, 'Could not insert before delimiter: ' + delimiter
|
||||
idx = data.rindex(delimiter)
|
||||
return data[:idx] + ins + data[idx:]
|
||||
|
||||
|
||||
def build_url(parts: List[str]) -> str:
|
||||
''' Build URL similar to lektor.utils.build_url '''
|
||||
url = ''
|
||||
for comp in parts:
|
||||
txt = str(comp).strip('/')
|
||||
if txt:
|
||||
url += '/' + txt
|
||||
if '.' not in url.rsplit('/', 1)[-1]:
|
||||
url += '/'
|
||||
return url or '/'
|
||||
|
||||
|
||||
class cached_property(Generic[T]):
|
||||
''' Calculate complex property only once. '''
|
||||
|
||||
def __init__(self, fn: Callable[[Any], T]) -> None:
|
||||
self.fn = fn
|
||||
|
||||
def __get__(self, obj: object, typ: Union[type, None] = None) -> T:
|
||||
if obj is None:
|
||||
return self # type: ignore
|
||||
ret = obj.__dict__[self.fn.__name__] = self.fn(obj)
|
||||
return ret
|
||||
287
lektor_groupby/vobj.py
Normal file
287
lektor_groupby/vobj.py
Normal file
@@ -0,0 +1,287 @@
|
||||
from lektor.build_programs import BuildProgram # subclass
|
||||
from lektor.context import get_ctx
|
||||
from lektor.db import _CmpHelper
|
||||
from lektor.environment import Expression
|
||||
from lektor.sourceobj import VirtualSourceObject # subclass
|
||||
from typing import (
|
||||
TYPE_CHECKING, List, Any, Dict, Optional, Generator, Iterator, Iterable
|
||||
)
|
||||
from .pagination import PaginationConfig
|
||||
from .pruner import VirtualPruner
|
||||
from .query import FixedRecordsQuery
|
||||
from .util import most_used_key, insert_before_ext, build_url, cached_property
|
||||
if TYPE_CHECKING:
|
||||
from lektor.pagination import Pagination
|
||||
from lektor.builder import Artifact
|
||||
from lektor.db import Record
|
||||
from .config import Config
|
||||
|
||||
VPATH = '@groupby' # potentially unsafe. All matching entries are pruned.
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# VirtualSource
|
||||
# -----------------------------------
|
||||
|
||||
class GroupBySource(VirtualSourceObject):
|
||||
'''
|
||||
Holds information for a single group/cluster.
|
||||
This object is accessible in your template file.
|
||||
Attributes: record, key, key_obj, slug, children, config
|
||||
'''
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
record: 'Record',
|
||||
key: str,
|
||||
config: 'Config',
|
||||
page_num: Optional[int] = None
|
||||
) -> None:
|
||||
super().__init__(record)
|
||||
self.__children = [] # type: List[str]
|
||||
self.__key_obj_map = [] # type: List[Any]
|
||||
self._expr_fields = {} # type: Dict[str, Expression]
|
||||
self.key = key
|
||||
self.config = config
|
||||
self.page_num = page_num
|
||||
|
||||
def append_child(self, child: 'Record', key_obj: Any) -> None:
|
||||
if child not in self.__children:
|
||||
self.__children.append(child.path)
|
||||
# __key_obj_map is later used to find most used key_obj
|
||||
self.__key_obj_map.append(key_obj)
|
||||
|
||||
def _update_attr(self, key: str, value: Any) -> None:
|
||||
''' Set or remove Jinja evaluated Expression field. '''
|
||||
# TODO: instead we could evaluate the fields only once.
|
||||
# But then we need to record_dependency() every successive access
|
||||
if isinstance(value, Expression):
|
||||
self._expr_fields[key] = value
|
||||
try:
|
||||
delattr(self, key)
|
||||
except AttributeError:
|
||||
pass
|
||||
else:
|
||||
if key in self._expr_fields:
|
||||
del self._expr_fields[key]
|
||||
setattr(self, key, value)
|
||||
|
||||
# -------------------------
|
||||
# Evaluate Extra Fields
|
||||
# -------------------------
|
||||
|
||||
def finalize(self, key_obj: Optional[Any] = None) \
|
||||
-> 'GroupBySource':
|
||||
# make a sorted children query
|
||||
self._query = FixedRecordsQuery(self.pad, self.__children, self.alt)
|
||||
self._query._order_by = self.config.order_by
|
||||
del self.__children
|
||||
# set indexed original value (can be: str, int, float, bool, obj)
|
||||
self.key_obj = key_obj or most_used_key(self.__key_obj_map)
|
||||
del self.__key_obj_map
|
||||
|
||||
if key_obj: # exit early if initialized through resolver
|
||||
return self
|
||||
# extra fields
|
||||
for attr in self.config.fields:
|
||||
self._update_attr(attr, self.config.eval_field(attr, on=self))
|
||||
return self
|
||||
|
||||
@cached_property
|
||||
def slug(self) -> Optional[str]:
|
||||
# evaluate slug Expression once we need it
|
||||
slug = self.config.eval_slug(self.key, on=self)
|
||||
if slug and slug.endswith('/index.html'):
|
||||
slug = slug[:-10]
|
||||
return slug
|
||||
|
||||
# -----------------------
|
||||
# Pagination handling
|
||||
# -----------------------
|
||||
|
||||
@property
|
||||
def supports_pagination(self) -> bool:
|
||||
return self.config.pagination['enabled'] # type: ignore[no-any-return]
|
||||
|
||||
@cached_property
|
||||
def _pagination_config(self) -> 'PaginationConfig':
|
||||
# Generate `PaginationConfig` once we need it
|
||||
return PaginationConfig(self.record.pad.env, self.config.pagination,
|
||||
self._query.total)
|
||||
|
||||
@cached_property
|
||||
def pagination(self) -> 'Pagination':
|
||||
# Generate `Pagination` once we need it
|
||||
return self._pagination_config.get_pagination_controller(self)
|
||||
|
||||
def __iter_pagination_sources__(self) -> Iterator['GroupBySource']:
|
||||
''' If pagination enabled, yields `GroupBySourcePage` sub-pages. '''
|
||||
# Used in GroupBy.make_once() to generated paginated child sources
|
||||
if self._pagination_config.enabled and self.page_num is None:
|
||||
for page_num in range(self._pagination_config.count_pages(self)):
|
||||
yield self.__for_page__(page_num + 1)
|
||||
|
||||
def __for_page__(self, page_num: Optional[int]) -> 'GroupBySource':
|
||||
''' Get source object for a (possibly) different page number. '''
|
||||
assert page_num is not None
|
||||
return GroupBySourcePage(self, page_num)
|
||||
|
||||
# ---------------------
|
||||
# Lektor properties
|
||||
# ---------------------
|
||||
|
||||
@property
|
||||
def path(self) -> str: # type: ignore[override]
|
||||
# Used in VirtualSourceInfo, used to prune VirtualObjects
|
||||
vpath = f'{self.record.path}{VPATH}/{self.config.key}/{self.key}'
|
||||
if self.page_num:
|
||||
vpath += '/' + str(self.page_num)
|
||||
return vpath
|
||||
|
||||
@cached_property
|
||||
def url_path(self) -> str: # type: ignore[override]
|
||||
''' Actual path to resource as seen by the browser. '''
|
||||
# check if slug is absolute URL
|
||||
slug = self.slug
|
||||
if slug and slug.startswith('/'):
|
||||
parts = [self.pad.get_root(alt=self.alt).url_path]
|
||||
else:
|
||||
parts = [self.record.url_path]
|
||||
# slug can be None!!
|
||||
if not slug:
|
||||
return build_url(parts)
|
||||
# if pagination enabled, append pagination.url_suffix to path
|
||||
if self.page_num and self.page_num > 1:
|
||||
sffx = self._pagination_config.url_suffix
|
||||
if '.' in slug.rsplit('/', 1)[-1]:
|
||||
# default: ../slugpage2.html (use e.g.: url_suffix = .page.)
|
||||
parts.append(insert_before_ext(
|
||||
slug, sffx + str(self.page_num), '.'))
|
||||
else:
|
||||
# default: ../slug/page/2/index.html
|
||||
parts += [slug, sffx, self.page_num]
|
||||
else:
|
||||
parts.append(slug)
|
||||
return build_url(parts)
|
||||
|
||||
def iter_source_filenames(self) -> Generator[str, None, None]:
|
||||
''' Enumerate all dependencies '''
|
||||
if self.config.dependencies:
|
||||
yield from self.config.dependencies
|
||||
for record in self.children:
|
||||
yield from record.iter_source_filenames()
|
||||
|
||||
# def get_checksum(self, path_cache: 'PathCache') -> Optional[str]:
|
||||
# deps = [self.pad.env.jinja_env.get_or_select_template(
|
||||
# self.config.template).filename]
|
||||
# deps.extend(self.iter_source_filenames())
|
||||
# sums = '|'.join(path_cache.get_file_info(x).filename_and_checksum
|
||||
# for x in deps if x) + str(self.children.count())
|
||||
# return hashlib.sha1(sums.encode('utf-8')).hexdigest() if sums else None
|
||||
|
||||
def get_sort_key(self, fields: Iterable[str]) -> List:
|
||||
def cmp_val(field: str) -> Any:
|
||||
reverse = field.startswith('-')
|
||||
if reverse or field.startswith('+'):
|
||||
field = field[1:]
|
||||
return _CmpHelper(getattr(self, field, None), reverse)
|
||||
|
||||
return [cmp_val(field) for field in fields or []]
|
||||
|
||||
# -----------------------
|
||||
# Properties & Helper
|
||||
# -----------------------
|
||||
|
||||
@property
|
||||
def children(self) -> FixedRecordsQuery:
|
||||
''' Return query of children of type Record. '''
|
||||
return self._query
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
# Used for virtual path resolver
|
||||
if key in ('_path', '_alt'):
|
||||
return getattr(self, key[1:])
|
||||
return self.__missing__(key)
|
||||
|
||||
def __getattr__(self, key: str) -> Any:
|
||||
''' Lazy evaluate custom user field expressions. '''
|
||||
if key in self._expr_fields:
|
||||
expr = self._expr_fields[key]
|
||||
return expr.evaluate(self.pad, this=self, alt=self.alt)
|
||||
raise AttributeError
|
||||
|
||||
def __lt__(self, other: 'GroupBySource') -> bool:
|
||||
# Used for |sort filter (`key_obj` is the indexed original value)
|
||||
if isinstance(self.key_obj, (bool, int, float)) and \
|
||||
isinstance(other.key_obj, (bool, int, float)):
|
||||
return self.key_obj < other.key_obj
|
||||
if self.key_obj is None:
|
||||
return False # this will sort None at the end
|
||||
if other.key_obj is None:
|
||||
return True
|
||||
return str(self.key_obj).lower() < str(other.key_obj).lower()
|
||||
|
||||
def __eq__(self, other: object) -> bool:
|
||||
# Used for |unique filter
|
||||
if self is other:
|
||||
return True
|
||||
return isinstance(other, GroupBySource) and \
|
||||
self.path == other.path and self.slug == other.slug
|
||||
|
||||
def __hash__(self) -> int:
|
||||
# Used for hashing in set and dict
|
||||
return hash((self.path, self.slug))
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return '<GroupBySource path="{}" children={}>'.format(
|
||||
self.path,
|
||||
self.children.count() if hasattr(self, 'children') else '?')
|
||||
|
||||
|
||||
# -----------------------------------
|
||||
# BuildProgram
|
||||
# -----------------------------------
|
||||
|
||||
class GroupByBuildProgram(BuildProgram):
|
||||
''' Generate Build-Artifacts and write files. '''
|
||||
|
||||
def produce_artifacts(self) -> None:
|
||||
pagination_enabled = self.source._pagination_config.enabled
|
||||
if pagination_enabled and self.source.page_num is None:
|
||||
return # only __iter_pagination_sources__()
|
||||
url = self.source.url_path
|
||||
if url.endswith('/'):
|
||||
url += 'index.html'
|
||||
self.declare_artifact(url, sources=list(
|
||||
self.source.iter_source_filenames()))
|
||||
|
||||
def build_artifact(self, artifact: 'Artifact') -> None:
|
||||
get_ctx().record_virtual_dependency(VirtualPruner())
|
||||
artifact.render_template_into(
|
||||
self.source.config.template, this=self.source)
|
||||
|
||||
|
||||
class GroupBySourcePage(GroupBySource):
|
||||
''' Pagination wrapper. Redirects get attr/item to non-paginated node. '''
|
||||
|
||||
def __init__(self, parent: 'GroupBySource', page_num: int) -> None:
|
||||
self.__parent = parent
|
||||
self.page_num = page_num
|
||||
|
||||
def __for_page__(self, page_num: Optional[int]) -> 'GroupBySource':
|
||||
''' Get source object for a (possibly) different page number. '''
|
||||
if page_num is None:
|
||||
return self.__parent
|
||||
if page_num == self.page_num:
|
||||
return self
|
||||
return GroupBySourcePage(self.__parent, page_num)
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
return self.__parent.__getitem__(key)
|
||||
|
||||
def __getattr__(self, key: str) -> Any:
|
||||
return getattr(self.__parent, key)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return '<GroupBySourcePage path="{}" page={}>'.format(
|
||||
self.__parent.path, self.page_num)
|
||||
148
lektor_groupby/watcher.py
Normal file
148
lektor_groupby/watcher.py
Normal file
@@ -0,0 +1,148 @@
|
||||
from typing import (
|
||||
TYPE_CHECKING, Dict, List, Any, Union, NamedTuple,
|
||||
Optional, Callable, Iterator, Generator
|
||||
)
|
||||
from .backref import VGroups
|
||||
from .model import ModelReader
|
||||
from .vobj import GroupBySource
|
||||
if TYPE_CHECKING:
|
||||
from lektor.db import Pad, Record
|
||||
from .config import Config
|
||||
from .model import FieldKeyPath
|
||||
|
||||
|
||||
class GroupByCallbackArgs(NamedTuple):
|
||||
record: 'Record'
|
||||
key: 'FieldKeyPath'
|
||||
field: Any # lektor model data-field value
|
||||
|
||||
|
||||
GroupingCallback = Callable[[GroupByCallbackArgs], Union[
|
||||
Iterator[Any],
|
||||
Generator[Any, Optional[GroupBySource], None],
|
||||
]]
|
||||
|
||||
|
||||
class Watcher:
|
||||
'''
|
||||
Callback is called with (Record, FieldKeyPath, field-value).
|
||||
Callback may yield 0-n objects.
|
||||
'''
|
||||
|
||||
def __init__(self, config: 'Config') -> None:
|
||||
self.config = config
|
||||
self._root = self.config.root
|
||||
|
||||
def grouping(self, flatten: bool = True) \
|
||||
-> Callable[[GroupingCallback], None]:
|
||||
'''
|
||||
Decorator to subscribe to attrib-elements.
|
||||
If flatten = False, dont explode FlowType.
|
||||
|
||||
(record, field-key, field) -> value
|
||||
'''
|
||||
def _decorator(fn: GroupingCallback) -> None:
|
||||
self.flatten = flatten
|
||||
self.callback = fn
|
||||
return _decorator
|
||||
|
||||
def initialize(self, pad: 'Pad') -> None:
|
||||
''' Reset internal state. You must initialize before each build! '''
|
||||
assert callable(self.callback), 'No grouping callback provided.'
|
||||
self._model_reader = ModelReader(pad.db, self.config.key, self.flatten)
|
||||
self._root_record = {} # type: Dict[str, Record]
|
||||
self._state = {} # type: Dict[str, Dict[Optional[str], GroupBySource]]
|
||||
self._rmmbr = [] # type: List[Record]
|
||||
for alt in pad.config.iter_alternatives():
|
||||
self._root_record[alt] = pad.get(self._root, alt=alt)
|
||||
self._state[alt] = {}
|
||||
|
||||
def should_process(self, node: 'Record') -> bool:
|
||||
''' Check if record path is being watched. '''
|
||||
return str(node['_path']).startswith(self._root)
|
||||
|
||||
def process(self, record: 'Record') -> None:
|
||||
'''
|
||||
Will iterate over all record fields and call the callback method.
|
||||
Each record is guaranteed to be processed only once.
|
||||
'''
|
||||
for key, field in self._model_reader.read(record):
|
||||
args = GroupByCallbackArgs(record, key, field)
|
||||
_gen = self.callback(args)
|
||||
try:
|
||||
key_obj = next(_gen)
|
||||
while True:
|
||||
if self.config.key_obj_fn:
|
||||
vobj = self._persist_multiple(args, key_obj)
|
||||
else:
|
||||
vobj = self._persist(args, key_obj)
|
||||
# return groupby virtual object and continue iteration
|
||||
if isinstance(_gen, Generator) and not _gen.gi_yieldfrom:
|
||||
key_obj = _gen.send(vobj)
|
||||
else:
|
||||
key_obj = next(_gen)
|
||||
except StopIteration:
|
||||
del _gen
|
||||
|
||||
def _persist_multiple(self, args: 'GroupByCallbackArgs', obj: Any) \
|
||||
-> Optional[GroupBySource]:
|
||||
# if custom key mapping function defined, use that first
|
||||
res = self.config.eval_key_obj_fn(on=args.record,
|
||||
context={'X': obj, 'ARGS': args})
|
||||
if isinstance(res, (list, tuple)):
|
||||
for k in res:
|
||||
self._persist(args, k) # 1-to-n replacement
|
||||
return None
|
||||
return self._persist(args, res) # normal & null replacement
|
||||
|
||||
def _persist(self, args: 'GroupByCallbackArgs', obj: Any) \
|
||||
-> Optional[GroupBySource]:
|
||||
''' Update internal state. Return grouping parent. '''
|
||||
if not isinstance(obj, (str, bool, int, float)) and obj is not None:
|
||||
raise ValueError(
|
||||
'Unsupported groupby yield type for [{}]:'
|
||||
' {} (expected str, got {})'.format(
|
||||
self.config.key, obj, type(obj).__name__))
|
||||
|
||||
if obj is None:
|
||||
# if obj is not set, test if config.replace_none_key is set
|
||||
slug = self.config.replace_none_key
|
||||
obj = slug
|
||||
else:
|
||||
# if obj is set, apply config.key_map (convert int -> str)
|
||||
slug = self.config.slugify(str(obj)) or None
|
||||
# if neither custom mapping succeeded, do not process further
|
||||
if not slug or obj is None:
|
||||
return None
|
||||
# update internal object storage
|
||||
alt = args.record.alt
|
||||
if slug not in self._state[alt]:
|
||||
src = GroupBySource(self._root_record[alt], slug, self.config)
|
||||
self._state[alt][slug] = src
|
||||
else:
|
||||
src = self._state[alt][slug]
|
||||
|
||||
src.append_child(args.record, obj)
|
||||
# reverse reference
|
||||
VGroups.of(args.record).add(args.key, src)
|
||||
return src
|
||||
|
||||
def remember(self, record: 'Record') -> None:
|
||||
self._rmmbr.append(record)
|
||||
|
||||
def iter_sources(self) -> Iterator[GroupBySource]:
|
||||
''' Prepare and yield GroupBySource elements. '''
|
||||
for x in self._rmmbr:
|
||||
self.process(x)
|
||||
del self._rmmbr
|
||||
for vobj_list in self._state.values():
|
||||
for vobj in vobj_list.values():
|
||||
yield vobj.finalize()
|
||||
# cleanup. remove this code if you'd like to iter twice
|
||||
del self._model_reader
|
||||
del self._root_record
|
||||
del self._state
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return '<GroupByWatcher key="{}" enabled={}>'.format(
|
||||
self.config.key, self.config.enabled)
|
||||
7
setup.py
7
setup.py
@@ -1,11 +1,11 @@
|
||||
from setuptools import setup
|
||||
|
||||
with open('README.md') as fp:
|
||||
with open('README.md', encoding='utf8') as fp:
|
||||
longdesc = fp.read()
|
||||
|
||||
setup(
|
||||
name='lektor-groupby',
|
||||
py_modules=['lektor_groupby'],
|
||||
packages=['lektor_groupby'],
|
||||
entry_points={
|
||||
'lektor.plugins': [
|
||||
'groupby = lektor_groupby:GroupByPlugin',
|
||||
@@ -13,7 +13,7 @@ setup(
|
||||
},
|
||||
author='relikd',
|
||||
url='https://github.com/relikd/lektor-groupby-plugin',
|
||||
version='0.9.2',
|
||||
version='0.9.8',
|
||||
description='Cluster arbitrary records with field attribute keyword.',
|
||||
long_description=longdesc,
|
||||
long_description_content_type="text/markdown",
|
||||
@@ -27,7 +27,6 @@ setup(
|
||||
'cluster',
|
||||
],
|
||||
classifiers=[
|
||||
'Development Status :: 5 - Production/Stable',
|
||||
'Environment :: Web Environment',
|
||||
'Environment :: Plugins',
|
||||
'Framework :: Lektor',
|
||||
|
||||
Reference in New Issue
Block a user