Pico 4.0 and beyond

[PhrozenByte]

This issue is intended to collect and discuss ideas about Pico 2.0 3.0 4.0, especially on how we can make Pico's plugin system more flexible than ever. The changes are very substantial and far reaching, therefore they can't make it into the soon-to-be-released Pico 2.0 3.0. The changes break BC, therefore they can't make it into Pico 1.1 2.1 3.1, but Pico 2.0 3.0 4.0. Features planned for Pico 2.0 can be found in #270. Feedback is appreciated! 😃

General

• Move Pico to the \picocms\Pico namespace and use a PSR-4 autoloader. Use PicoDeprecated to also provide all classes in the root namespace (using class_alias()), otherwise this would break all existing plugins.
• Add Unit Tests.
• Split Pico::run() into multiple public methods ("phases"), but still call them through Pico::run() (init, request URL, loading contents, evaluating contents (YAML + Markdown), page discovery, page rendering (Twig)).
• Use PicoMarkdownParser, PicoYamlParser and PicoTemplateEngine instead of \Parsedown, \Symfony\Component\Yaml\Parser and \Twig_Environment as type hints and add appropriate wrapper classes for Parsedown, Symfony YAML and Twig, allowing one to extend/replace them completely.
  • Enable some useful Symfony YAML flags by default (like converting keys to string automatically).
  • Also see cancelled "replace YAML/Parsedown/Twig" ToDo below.
• Drop all remaining pre-v1.0 behavior from PicoDeprecated, also remove v1.0+ behaviour with a notable performance impact
• Think about how users can easily install plugins with dependencies when they aren't using composer, but a pre-bundled release
• Think about refactoring Pico's plugin discovery process in general: having a lot of (partially duplicate) code for filesystem-based and composer-based plugins is bad, furthermore there are issues with the current system like Error running Pico on Synology Diskstation #538; maybe use extensible iterators?
• Allow theme developers to register meta headers and change Twig's default config (maybe using a config.yml in the theme's dir?) Implemented with Pico 2.1
• Allow theme developers to register static pages in Pico's pico-theme.yml (e.g. a virtual contact page is added to the pages list not requiring a contact.md with e.g. Template: contact to exist)
• Support relative page URLs in Markdown files
  • We must likely hook into Parsedown and make them absolute
  • Relative paths to other files pages (e.g. assets/image.jpg) should be supported, too
  • Relative paths must be interpreted so that content dir breakouts are stripped, i.e. ../page from content/sub/page.md ends up as https://example.com/pico/page, whereas ../../assets/index.jpg ends up as https://example.com/pico/assets/index.jpg (instead of https://example.com/assets/index.jpg)

Page management

• Implement lazy loading using page objects (i.e. use simple objects instead of $page arrays)
  • Performance, Performance, Performance
  • Implement ArrayAccess; not only to maintain BC, but also to make the object's data easier to use (especially for non-experienced developers)
    • Also switch a page's meta data to ArrayAccess objects and allow case insensitive keys (i.e. it no longer matters whether a meta value is Title or title, one can always access it with Title, title, or even tItLe)
    • Only propagate title, date, template and hidden as page data; everything else (e.g. description, author, robots) gets ordinary YAML metadata and can be accessed via page.meta
  • Pico initially loads no file contents, only page.id and page.url are available at first; as soon as something else is accessed, the file's contents are read; depending on what has been accessed, Pico processes either the YAML frontmatter or the Markdown contents (i.e. $page['content'] won't be deprecated anymore).
  • Plugin developers should be able to implement dynamic values with callbacks (e.g. PicoPage::addDynamicValue(string $key, callable $callback)). This also allows plugin developers to implement lazy loading for custom values.
  • Open question: Should we still allow regular page arrays in the $pages array? Probably not: plugin developers will have to differentiate both cases otherwise, what makes the whole feature a pain in the ass for them. Thus we need a conversion method, otherwise we would break all existing plugins...
  • Page objects don't know their respective next and previous pages, but parent and child pages (see below)
• Implement a lazy page tree (using iterable objects) as better performing alternative to the regular pages array
  • Performance, Performance, Performance!
    • Rather than always loading the whole page tree (a Traversable, ArrayAccess object like the pages array, see above), nothing is loaded by default.
    • When accessing the page tree for the first time, Pico discovers only pages directly inside content/ (with the second level of lazy loading as elucidated above). Pages in sub-directories are accessible through a children key. However, the contents of directories aren't discovered until they are explicitly requested by accessing said children key.
    • Most themes build their page menu by iterating over pages on the first level anyway. This allows Pico to discover only the pages it needs to know, i.e. just the pages directly inside the content/ dir and without any sub directory. This should heavily improve performance when a Pico instance is supposed to serve hundreds or thousands of pages.
    • Even the regular pages array is actually empty in the beginning. However, by accessing the variable (i.e. by iterating over it or by accessing a key) the whole page tree is being loaded (what shatters our efforts). We should encourage plugin/theme developers to use the page tree instead.
  • Pico manages a global sorted page tree
    • Drop support for sorting the pages list globally and rather use concomitant alphabetical sorting; i.e. Pico first discovers content/index.md only, then all other pages in content (e.g. content/foo.md) and sorts them alphabetically, then checks for a content/foo/ directory and discovers all pages in it (e.g. content/foo/bar.md), etc. and slices the just discovered pages in
  • Drop support for the $pages variable and force users to use the pages() function
    • Open question: PicoDeprecated should implement it as some magic variable to still support lazy loading (is this even possible?)
    • The pages() function should return an object instead of an array with sortable page objects wrapping the requested page objects; the pages list object can then be sorted and set a page's respective previous and next page to the page wrapper
      • It should be possible to perform both shallow and recursive sorting
  • Open question: How do we make this backwards-compatible with PicoDeprecated?
    • Use ArrayObject or ArrayIterator instead of a regular $pages array. This allows us to convert page arrays to objects as soon as they are added.
    • However, PHP's built-in array functions (e.g. array_keys()) won't work anymore... According to that we must pass a regular array to the onPagesLoaded event for older plugins, otherwise we would pretty likely break many of them. PicoDeprecated could then iterate through the $pages array and convert them appropriately.
    • Use event system versioning?
  • Refactor the pages() function to use offset and length parameters instead of depth, depthOffset and offset; see Show (grand)parent-page title #627 (comment) and earlier comments
    • Utilize theme API versioning to actually drop depth and depthOffset, but use PicoDeprecated to inject the old function for old themes
    • Thinking about using API versioning anyway, we could also rename length to depth - it better reflects what it does... But it's not to confuse with the "old" depth
    • Taking page objects into consideration we should use those in most cases, i.e. there should be keys to specifically not just access a page's child pages, but ancestors until a user-defined depth

Plugin event system

• Don't trigger all events on all plugins. Let plugins register the events they want to use instead. This heavily increases performance with a large number of plugins, because method_exists calls are comparatively expensive compared to a simple foreach per event.
  • Open question: Either introduce a new onSinglePluginLoaded event or a new PicoPluginInterface::getEvents() method.
    • The latter breaks BC, but with this new approach, we must refactor AbstractPicoPlugin::handleEvent() anyway, therefore we probably can't implement it without breaking BC one way or the other.
    • We can circumvent this by letting \AbstractPicoPlugin and \picocms\Pico\AbstractPlugin differ in functionality (i.e. the first mentioned implements the getEvents() method in a BC way by examining a ReflectionClass of the plugin, or by letting \PicoPluginInterface lack the getEvents() method entirely).
• Allow plugins to return false on preliminary events (e.g. onContentLoading) to prevent Pico from performing a specific processing step (Pico::run() skips Pico::loadFileContent()). Returning true or null works as with Pico 1.0 and changes nothing. The subsequent event is still triggered (onContentLoaded), but the payload variable ($rawContent) is empty. The event is triggered with special priority on this plugin (regardless of the regular processing order), so it can set the variable before any other plugin receives the event.
  • Example:
    • A markdown cache plugin returns false during onMetaParsing and onContentParsing to load both meta data and the parsed contents from its cache.
  • Affected Events:
    • onContentLoading (completely skips on404Content… events) and onContentLoaded
    • on404ContentLoading and on404ContentLoaded
    • onMetaParsing and onMetaParsed
    • onContentParsing (simulates onContentPrepared), onContentPrepared and onContentParsed
    • onPagesLoading (onSinglePage… events will be simulated) and onPagesLoaded
    • onSinglePageLoading (new event) and onSinglePageLoaded
    • onPageRendering and onPageRendered
• Allow plugins to return false on the onRequestUrl or onRequestFile events to completely skip Pico's processing. The only remaining event to trigger is onOutput (new event that is only triggered when Pico's processing is skipped) right before Pico returns $output.
  • Example:
    • A static HTML cache plugin returns false during onRequestFile, bypasses Pico's processing completely and returns the cached contents during onOutput.

New official plugins

• Markdown cache
• Static HTML cache
  • Save rendered output of pages to static HTML files
  • Rely on OS to detect file changes (last modification time of .md files)
  • Add Cache: No meta header to prevent pages from being cached
  • Add a event to let plugins "register" non-content pages for caching (should be triggered right after onConfigLoaded to allow plugins to change their behavior when caching is requested)
  • Explicitly allow combining statically cached and dynamic pages
  • Open questions
    • What happens when a page is added (i.e. page navigation changes)?
    • What happens when a plugin or theme is added/updated/removed?
    • How to determine all URLs that need to be parsed? Markdown files don't necessarily have a 1:1 relation to pages, just think of collections or hidden meta files
      • Ignore files and directories starting with a _
      • Allow users to explicitly specify the URL of a page
  • Use this feature to allow Pico to act as a static website generator (allow plugins to distinct between "static HTML cache" and "static website generator" mode)
    • Use Pico (with Travis, PHP's development server and wget -r) for our website rather than Jekyll
  • Plugin plugins (:smile:):
    • Add a URLs sequence meta header to support alternative URLs (like Jekyll's Redirect From plugin
• Search (using Lucene?)
  • Problem: How to determine the URL of a found Markdown file? Markdown files don't necessarily have a 1:1 relation to pages, just think of collections or hidden meta files
  • Possible solution: Use a static HTML cache and search in the HTML files
  • Possible solution: Do the exact same things as the static HTML cache (see above)
• Multilanguage (i18n)
  • Contents: use language-specific content directories or file extensions?
  • Themes: use PHP intl and Twig_Extensions_Extension_Intl
  • References: How to create multilingual web site? #335
• Performance statistics: See Pico 4.0 and beyond #317 (comment)
• Import plugins to import contents of other CMS (e.g. WordPress, see https://github.com/gilbitron/WordPress-to-Pico-Exporter)
  • Use a HTML to Markdown converter?
• Data Files
  • Support independent (meta) data files (e.g. content/catalog.yml)
  • The files are accessed similar to pages (e.g. {{ data.catalog }}).
  • A data file named after a markdown file (e.g. both content/catalog.yml and content/catalog.md exist) is non-recursively merged into the page's meta data (i.e. into {{ pages.catalog.meta }}). However, the YAML frontmatter takes preference and the data file can still be accessed via {{ data.catalog }}.
  • The same happens for all pages (non-recursive) in a directory if there's a data file with the same name as the directory (e.g. _collection.yml and _collection/ directory). You can enforce recursion for e.g. _collection/subdir/ by creating _collection/subdir.yml.
• Redirect pages (like Jekyll's Redirect From plugin)

Not planned anymore

• Allow a single plugin to hook into Pico to basically replace YAML/Parsedown/Twig with something different. Rather than hooking into the instantiation of \Symfony\Component\Yaml\Parser in Pico::parseFileMeta(), Pico::registerParsedown() and/or Pico::registerTwig(), it should be possible to replace the Pico::parseFileMeta() method, the Pico::prepareFileContent()/Pico::parseFileContent() methods (+ the markdown Twig filter in PicoTwigExtension) and/or the call of Pico::$twig->render(). Otherwise the plugin needs to re-implement the internal structures and workings of the YAML parser/Parsedown/Twig, what isn't desirable. I'm currently not sure about how this interacts with the $twig parameter of the onPageRendering event (maybe drop the parameter and add a new onTwigRegistered event?). The plugin needs to be registered explicitly in config/config.php to work.
  • Example (quite a stretch):
    • Instead of parsing Markdown, parse MediaWiki syntax.

[Crank1d]

I have idea on how to make pages more flexible - in other words, having "custom page types" based on XML schema file (or YAML, maybe?) in content directory without writing specific meta data into every .md file.

This would make Pico more CMS-like and structured, so for example, if someone would like to have "blog" or "products" type of pages, each containing alot of "child pages" (subpages), he should create XML schema file called "blogposts.xml", which contains info/data describing how subpages should behave, like order of pages, pagination count etc. and any pages associated with schema put into subfolder in content directory. This would probably branch content directory structure into several subfolders for specific page type so structure would be something like "default" - for default pages, "posts" - for blog posts or any other page types. Index pages for those page types could also contain page loop/query of all of its child pages - just like blog has, similar to Wordpress but still being simple.

Also, maybe having another XML file for post categories which would contain something like page IDs under specific category section ("Animals", for example) and when category URL is requested, category page would do the loop and display all pages associated with this category page.

I think this would be possible but would require to rewrite core Pico files, since creating Blog plugin alone would not have access to way/order pages are displayed since Pico searches folders recursively for .md files in content directory.

This would increase complexity of Pico but it would be a welcome feature which could possibly draw more people to use Pico for their website, especially those having smaller blog dedicated to specific subject, like personal travel blog, for example.

[PhrozenByte]

Thank you for your feedback @Crank1d! 👍

I did not really understand what data you're suggesting to store in these XML files. Pico leaves page presentation (i.e. how a page in a specific folder (e.g. blog) looks an behaves) completely up to themes. For example, @smcdougall's NotePaper theme interprets content files differently based on YAML headers (e.g. NotePaper's widget feature or blog articles). NotePaper specifically also allows you to set a Tag meta header and lists all pages containing a specific tag (so implementing categories shouldn't be a problem either).

As far as I understand your suggestion right, Pico already allows you to do something like this - it just doesn't require you to create a separate XML file, you can simply store that information in the YAML header of the content file.

[Crank1d]

Something like "Page Blueprints", implemented in GRAV - http://learn.getgrav.org/forms/blueprints

Scheme file would contain custom defined meta data for all its child pages/subpages, so there would be no need to define custom meta from plugin.

Maybe Im wrong, but Pico currently doesnt support "parent-children" relationship of pages?

[PhrozenByte]

As far as I understand Grav's documentation right, blueprints are about metadata for plugins and themes, not about pages. Pico has no administration backend. We're indeed working on a editor plugin for Pico 1.1 (see PhrozenByte/pico-admin), but that's rather a "enhanced" text input field. Much more "administration backend" is in contradiction to Pico's "stupidly simple" goal (what isn't limited to the user-side, but also Pico's sourcecode).

Pico doesn't have a "parent-child" relationship of pages in the narrow sense. When it comes down to "folder-specific meta data defaults": Pico absolutely allows you to do something like this, e.g. by creating a _meta.md file and adding its meta data to all other pages in this directory by using the onPagesLoaded event with a very simple plugin. Apart from that, Pico leaves the directory structure completely up to the user and maybe a custom theme that interprets the directory structure to do something special (like listing all pages in categories).