logo logo
Development: Structure

Files in the Securepub repository play one of three roles:

  • Command and control
  • Source code and content
  • Workspace project compilation

Command and control

Securepub's top level repository directory contains "command and control" files:

  • the project LICENSE and README.md
  • dotfiles with development workflow operations
  • the local.sh script

The top level directory is kept clean of package.json and other clutter.
These stay under work/ so the typical Securepub git working tree ↗︎ looks like

├── .envrc                      ⎞
├── .envrc.default              ⎥ command and control files
├── .gitignore                  ⎥
├── .gitlab-ci.yml              ⎥
├── .justfile                   ⎥
├── .procfile                   ⎥
├── LICENSE                     ⎥
├── README.md                   ⎥
├── local.sh                    ⎠
├── src                         ⎞ source and work areas
└── work                        ⎠
  • LICENSE and README.md are self-explanatory.

  • .envrc tells direnv where executables and tools are so that their directories may be added to the shell path.

  • .gitignore specifies files and build artifacts not managed by git.

  • GitLab CI/CD executes the .gitlab-ci.yml file when commits are pushed to the GitLab project repository.

  • .justfile defines commands for just ↗︎. This is the most important file as it defines the top-level commands used in typical development tasks such as:

    • % just dev (or % just without any arguments) sets up the development environment and starts processes listed in the .procfile which incrementally rebuild parts of the Securepub site whenever a source file is changed.

    • % just build sets up the development environment, builds the local and local-s origin directories and adds them to the sp.zip file.

    • % just clean removes all build directories and artifacts.

    • % just veryclean does the work of % just clean and also removes all downloaded packages, node module directories, elm stuff directories and python virtual environments used during development.

  • .procfile is a Procfile ↗︎ with processes Overmind ↗︎ runs during incremental development as described above.

  • local.sh serves the local and local-s directories. It is also bundled in the sp.zip file built by % just build.

The article Introducing Overmind and Hivemind ↗︎ has a good introduction to how they work.

Source code and content

The src/ directory holds Securepub's Zola source pages and its Elm and Typescript modules.

There are

  • 19 numbered subdirectories, one for each section of Securepub.
  • 3 unnumbered subdirectores for things used by multiple sections:
    • src/elm - 27 shared Elm modules
    • src/ts - 18 shared Typescript modules
    • src/workflow - 5 shared workflow definitions

Each section contains one more more of the Securepub site's pages.

The numbered sections roughly correspond to the site's routes. For example the 03.design section defines the pages and logic for pages whose URLs begin with the _v1.2401_/design prefix, the 41.member section has the pages withh the _v1.2401_/member prefix and so forth.

Every subdirectory of src/ has a _sync.conf file described in the synchronization documentation which specifies how files are copied into the work/ area.

A subset of src/ is shown below:

├── 00.test                     for testing/debugging
├── 01.welcome                  ⎞
├── 02.how-it-works             ⎥ documentation sections
├── 03.design                   ⎥
├── 04.development              ⎥
├── 05.deployment               ⎥
├── 06.about                    ⎥
├── 07.license                  ⎠
├── 10.create                   ⎞
├── 11.profile                  ⎥ application sections
├── 20.bundlelist               ⎥
├── 21.bundle                   ⎥
├── 30.topiclist                ⎥
├── 31.topic                    ⎥
├── 32.review                   ⎥
├── 40.memberlist               ⎥
├── 41.member                   ⎥
├── 60.join                     ⎥
├── 61.signin                   ⎠
├── elm
│   │
│   ├── _sync.conf              specifies Elm files copied to
│   │                           the work directory by .sync.py
│   │
│   ├── Auth                    ⎞
│   ├── BundleContents          ⎥ Elm modules used by the
│   ├── BundleData              ⎥  application sections
│   ...
│   ├── Userbase                ⎥
│   └── Visited                 ⎠
├── ts
│   │
│   ├── _sync.conf              specifies Typescript files copied
│   │                           to the work directory by .sync.py
│   │
│   ├── auth                    ⎞
│   ├── db                      ⎥ Typescript modules used
│   ├── dbitems                 ⎥  by the application sections
│   ...
│   ├── util                    ⎥
│   └── uw                      ⎠
└── workflow

Section structure

In addition to _sync.conf, each section has one or more of these directories:

  • <section>/elm - Elm module providing the section's interactive views.
  • <section>/ts - module which executes Typescript workflows for Elm.
  • <section>/workflow - diagnostic flowcharts for Typescript workflows.
  • <section>/zola - Markdown files Zola uses to generate static content.

For example here is the tree for the 11.profile section:

├── _sync.conf                    synchronization logic for section
├── elm
│   ├── Profile.elm               ⎞
│   └── Profile                   ⎥ Elm module managing the views
│       ├── Decode.elm            ⎥  which appear in this section
│       ├── Model.elm             ⎥
│       └── View.elm              ⎠
├── ts
│   ├── debug.ts                  ⎞ Typescript module which
│   ├── io.ts                     ⎥  provides Elm with data and
│   └── main.ts                   ⎠  runs workflows for Elm
├── workflow
│   ├── add_profile_load.yaml     ⎞ Yaml files describing workflows
│   ├── add_profile_run.yaml      ⎥  Typescript runs when section
│   ├── update_profile_load.yaml  ⎥  is loaded and when Elm issues
│   └── update_profile_run.yaml   ⎠  requests
└── zola
    ├── _index.md                 ⎞
    ├── _index.sp.md              ⎥  Markdown and yaml describing
    ├── _profile.yaml             ⎥   static content of the section
    ├── add.md                    ⎥
    └── add.sp.md                 ⎠

Application sections generally have all the subdirectories whereas documentation sections only have zola.

Working area

The work/ directory is the working area for compilers and bundlers which generate static HTML and Javascript files from the source code and content. The working area contains the workspace projects, their package.json files and the various temporary directories like node_modules and elm-stuff for the build tooling. It also contains static assets and templates Zola uses when building the site.

The files and directories of the work/ area play one of 6 roles:

  • Workspace setup, synchronization, patches and vendored modules.
  • Building the Elm module for the main Securepub site.
  • Building the Typescript modules for the main Securepub site.
  • Building the Typescript modules for the secure origin.
  • Building static HTML for the main Securepub site with Zola.
  • Building static HTML for the secure origin with Zola.

Workspace setup, synchronization, patches and vendored modules

These files deal with things related to the overall project and third-party code.

├── .sync.py                    ⎞ files for python synchronizer
├── pylintrc                    ⎠  and code generator
├── pnpm-workspace.yaml         pnpm workspace
├── package.json                ⎞ patch files for npm modules
├── patches                     ⎠
├── vendor-j2e                  ⎞ vendored json-to-elm and
└── vendor-jsp                  ⎠  javascript-playgrounds
  • .sync.py (and its pylintrc) provide the synchronization and code generation system which copies files from subdirectories of src/ to subdirectories of work/.

  • pnpm-workspace.yaml lists all the workspace packages managed by pnpm ↗︎.

  • package.json and the files in the patches directory apply minor changes to npm modules so that they work properly with Securepub.

  • work/vendor-j2e holds a stripped-down version of json-to-elm ↗︎ by Alex Korban used to generate Elm flags decoders for Securepub. The .sync.py process runs this utility.

  • work/vendor-jsp holds a stripped-down version of javascript-playgrounds ↗︎ by Devin Abbot sans embellishments (phone frames, python, typescript, etc) used to configure and interpret content bundle settings in the 21.bundle section of Securepub.

Building the Elm module for the main Securepub site

These files combine and compile the Elm modules into the _v1.2401_elm.js file with the Javascript code needed to render all the Elm views.

└── elm                         where elm code is compiled
    ├── elm.ts                  ⎞ ts interface for Elm type
    ├── index.ts                ⎠
    ├── package.json            how pnpm runs elm
    ├── elm.json                elm dependencies
    ├── _elm.sh                 ⎞ script to post-process the
    ├── _elm.awk                ⎠ Javascript generated by Elm
    └── src
        └── ...                 elm sources are copied here
  • elm.ts and index.ts provide the Typescript interface specifying the Elm port send and subscribe functions.

  • package.json identifies the Elm dependency and specifies the scripts run by pnpm.

  • elm.json identifies the Elm modules Securepub uses.

  • _elm.awk is called by _elm.sh to create a short bit of code to render a list of Dom nodes passed to Elm instead of just a single node. The code is patched into the generated Javascript with ex.

  • _elm.sh has the logic to run elm-live or elm make and depending on context and adjust the output Javascript with _elm.awk so that the resulting Elm code will render views into multiple dom elements from a single model. The final _v1.2401_elm.js it generates is written to the Zola static directory.

  • work/elm/src contains the files copied and generated by the _sync.py synchronization script which serve as input to elm-live and elm make.

Building the Typescript modules for the main Securepub site

These files combine and compile the Typescript modules into the _v1.2401_main.js and _v1.2401_uw.js files which allow Elm to interoperate with Userbase.

├── ts-configs                  shared typescript configuration
│   └── ...
├── auth                        ⎞
├── db                          ⎥ pnpm workspace packages
├── dbitems                     ⎥ for typescript modules.
├── dbmods                      ⎥
├── dbsources                   ⎥ each of these has
├── dbviews                     ⎥ package.json and tsconfig.json
├── entries                     ⎥
├── msg                         ⎥
├── progress                    ⎥
├── rcache                      ⎥
├── settings                    ⎥
├── share                       ⎥
├── types                       ⎥
├── util                        ⎠
├── uw                          userbase web worker module
└── main                        Securepub main module

shared ts-configs

  • work/ts-configs/ contains a collection of Typescript configuration files organized by the corresponding section of the tsconfig ↗︎ documentation. The various tsconfig.json in each Securepub module extend these files, occasionally overriding compilerOptions for their situation.

internal modules

Each of the following Typescript modules is copied by .sync.py from src/ according to the specified directive.

  • work/auth is used by each Securepub section to parse invitation links and authenticate the currently logged in user.
sync from src/auth/
src/ts/_sync.conf#L3-6 ↗︎
sync from src/auth/ src/ts/_sync.conf#L3-6 ↗︎
  • work/db is the "lowest" level of the data path.
    All code calling into Userbase uses this layer.
    It contains the DbUserbase wrapper for the Userbase API, the Args utilities to simplify providing proper arguments to Userbase functions and Services providing Effect-ts ↗︎ wrappers for DbUserbase.
sync from src/db/
src/ts/_sync.conf#L8-12 ↗︎
sync from src/db/ src/ts/_sync.conf#L8-12 ↗︎
  • work/dbitems contains Zod ↗︎ schema defintions. There is one Typescript file for each kind of Userbase database in Securepub. Zod schemas validate every item read or written to Userbase.
sync from src/dbitems/
src/ts/_sync.conf#L14-25 ↗︎
sync from src/dbitems/ src/ts/_sync.conf#L14-25 ↗︎
  • work/dbmods provides basic CRUD and permission management operations for each kind of Userbase database in Securepub.
sync from src/dbmods/
src/ts/_sync.conf#L27-44 ↗︎
sync from src/dbmods/ src/ts/_sync.conf#L27-44 ↗︎
  • work/dbsources tracks the results from asynchronous Userbase queries, maintains maps of objects and propagates updates to downstream sources and views.
sync from src/dbsources/
src/ts/_sync.conf#L46-63 ↗︎
sync from src/dbsources/ src/ts/_sync.conf#L46-63 ↗︎
  • work/dbviews aggregate data from db sources and send the results to Elm.
sync from src/dbviews/
src/ts/_sync.conf#L65-81 ↗︎
sync from src/dbviews/ src/ts/_sync.conf#L65-81 ↗︎
  • work/entries encodes and decodes the entries files with bundle Zip file metadata.
sync from src/entries/
src/ts/_sync.conf#L83-89 ↗︎
sync from src/entries/ src/ts/_sync.conf#L83-89 ↗︎
  • work/msg deals with parsing, serialization and session storage caching of JSON messages sent to Elm.
sync from src/msg/
src/ts/_sync.conf#L96-98 ↗︎
sync from src/msg/ src/ts/_sync.conf#L96-98 ↗︎
  • work/progress holds the Progress module which runs Effect workflow steps and reports their status to Elm via progress messages.
sync from src/progress/
src/ts/_sync.conf#L100-102 ↗︎
sync from src/progress/ src/ts/_sync.conf#L100-102 ↗︎
  • work/rcache holds the RCache (review cache) module and Rc interface which initialize the browser cache for the secure origin hosting the content of the review bundles.
sync from src/rcache/
src/ts/_sync.conf#L104-107 ↗︎
sync from src/rcache/ src/ts/_sync.conf#L104-107 ↗︎
  • work/settings holds the Settings interface to reads and write information to the javascript playground iframe on the bundle settings page.
sync from src/settings/
src/ts/_sync.conf#L111-115 ↗︎
sync from src/settings/ src/ts/_sync.conf#L111-115 ↗︎
  • work/share adjusts Userbase permissions on bundle and topic databases when members change who is allowed to access content or participate in a discussion.
sync from src/share/
src/ts/_sync.conf#L117-120 ↗︎
sync from src/share/ src/ts/_sync.conf#L117-120 ↗︎
  • work/types declares Typescript interfaces.
sync from src/types/
src/ts/_sync.conf#L125 ↗︎
sync from src/types/ src/ts/_sync.conf#L125 ↗︎
  • work/util provides common error handling, parsing and conversions.
sync from src/util/
src/ts/_sync.conf#L127-133 ↗︎
sync from src/util/ src/ts/_sync.conf#L127-133 ↗︎

main builds _v1.2401_main.js

  • work/main builds the top-level Main module which runs in the main origin and browser thread. The scripts section of work/main/package.json has the module's esbuild commands.

    The _sync.conf file of every section under src/ copies sources from the section's ts/ subdirectory into a subdirectory of work/main/.
sync from src/main/
src/ts/_sync.conf#L92-94 ↗︎
sync from src/main/ src/ts/_sync.conf#L92-94 ↗︎

uw builds _v1.2401_uw.js

  • work/uw builds the Userbase web worker used to perform Userbase permission operations as another Userbase user while the current user remains signed in from the main thread. The scripts section of work/uw/package.json has the module's esbuild commands.
sync from src/uw/
src/ts/_sync.conf#L135 ↗︎
sync from src/uw/ src/ts/_sync.conf#L135 ↗︎

Building the Typescript modules for the secure origin

These files combine and compile the Typescript modules into the _v1.2401_sw.js and _v1.2401_review.js files which serve the engagement content to users from the secure origin.

├── sw-s                        secure origin service worker module
└── review-s                    secure origin review module

These modules are copied by .sync.py from src/ according to the specified directive.

  • work/sw-s builds the service worker used in the secure origin to serve review content.
sync from src/sw-s/
src/ts/_sync.conf#L122-123 ↗︎
sync from src/sw-s/ src/ts/_sync.conf#L122-123 ↗︎
  • work/review-s builds the top-level module which runs in the secure origin's main thread.
sync from src/review-s/
src/ts/_sync.conf#L109 ↗︎
sync from src/review-s/ src/ts/_sync.conf#L109 ↗︎

Building static HTML for the main Securepub site with Zola

work/zola generates the static HTML files for the main Securepub site.

└── zola                        where zola is run
    ├── package.json            ⎞ how pnpm runs zola
    ├── _zola.sh                ⎠
    ├── prettierrc.json         ⎞ prettier config to validate
    ├── .prettierignore         ⎠ generated html files
    ├── translations.toml       ⎞ zola translations and configs
    ├── conf                    ⎥ combined into config.toml
    │   └── ...                 ⎠
    ├── content                 ⎞
    │   ├── _index.md           ⎥ zola content files
    │   ├── _index.sp.md        ⎥
    │   └── _v1.2401_           ⎥
    │       └── ...             ⎠
    ├── static                  ⎞ zola static assets
    │   └── _v1.2401_           ⎥  .js output files
    │       └── ...             ⎠
    └── templates               ⎞ zola templates and shortcodes
        └── shortcodes          ⎠  used by markdown content
  • package.json specifies the prettier dependency and specifies the scripts run by pnpm.

  • _zola.sh generates Zola's config.toml by merging a specified conf/ file and translations.toml and then launches zola build, prettier and/or zola serve as appropriate.

  • prettierrc.json and .prettierignore tell prettier what configuration it should use and what files to ignore for Securepub.

  • translations.toml contain the Zola translations used for the Securepub menu.

  • the work/zola/conf directory holds different base Zola configurations for each build environment.

  • the work/zola/content directory contains the files copied and generated from src/which build the final site.

  • the work/zola/static directory contains static assets and the final generated Javascript files from Typescript and Elm.

The Securepub pages section and the the Zola Overview ↗︎ provide more details.

Building static HTML for the Securepub secure origin with Zola

work/zola-s generates the static HTML files for the secure origin Securepub uses to isolate engagement content from Securepub's own collaboration pages.

The secure origin is a minimal site which runs some bootstrap code which communicates with the browser's main thread to initialize the browser cache and service worker with the engagement content before navigating to the appropriate topic URL.

└── zola-s
    ├── package.json            ⎞
    ├── _zola-s.sh              ⎥ zola configuration for
    ├── prettierrc.json         ⎥  secure origin site
    ├── conf                    ⎥
    ├── content                 ⎥
    ├── static                  ⎥
    └── templates               ⎥
        └── shortcodes          ⎠

The contents are much the same as described for the work/zola directory.
The scripts and config files differ in predictable ways.