From aa2ee001341bf26aa3891f3ba00b5bfdd307b513 Mon Sep 17 00:00:00 2001 From: "Reid D. 'arrdem' McKenzie" Date: Tue, 15 Feb 2022 10:22:53 -0700 Subject: [PATCH] config.toml -> pkg.toml, get cleanup working --- projects/cram/README.md | 109 +++++++++++++----- projects/cram/src/python/cram/__init__.py | 4 + projects/cram/src/python/cram/__main__.py | 18 ++- projects/cram/src/python/cram/common.py | 1 + projects/cram/src/python/cram/v1.py | 2 +- projects/cram/test.sh | 11 ++ .../hosts.d/test/{config.toml => pkg.toml} | 0 .../packages.d/p4/{config.toml => pkg.toml} | 0 .../packages.d/p6/{config.toml => pkg.toml} | 0 .../packages.d/p7/{config.toml => pkg.toml} | 0 .../default/{config.toml => pkg.toml} | 0 .../subpackage/{config.toml => pkg.toml} | 0 12 files changed, 116 insertions(+), 29 deletions(-) rename projects/cram/test/hosts.d/test/{config.toml => pkg.toml} (100%) rename projects/cram/test/packages.d/p4/{config.toml => pkg.toml} (100%) rename projects/cram/test/packages.d/p6/{config.toml => pkg.toml} (100%) rename projects/cram/test/packages.d/p7/{config.toml => pkg.toml} (100%) rename projects/cram/test/profiles.d/default/{config.toml => pkg.toml} (100%) rename projects/cram/test/profiles.d/default/subpackage/{config.toml => pkg.toml} (100%) diff --git a/projects/cram/README.md b/projects/cram/README.md index ab8888f..f1b4abb 100644 --- a/projects/cram/README.md +++ b/projects/cram/README.md @@ -4,61 +4,116 @@ An alternative to GNU Stow, more some notion of packages with dependencies and install scripts. -Think an Ansible or Puppet but anyarch and lite enough to check in with your dotfiles. +Think an Ansible, Puppet or even NixOS but anyarch and lite enough to check in with your dotfiles. ## Overview -Somewhat like Stow, Cram operates in terms of packages, which are directories with the following structure - +Cram operates on a directory of packages called `packages.d/`, and two directories of metapackages called `profiles.d` and `hosts.d`. -``` -/REQUIRES # A list of other packages this one requires. -/BUILD # 1. Perform any compile or package management tasks. -/PRE_INSTALL # 2. Any other tasks required before installation occurs. -/INSTALL # 3. Do whatever constitutes installation. - # This supercedes the default copying of files. -/POST_INSTALL # 4. Any cleanup or other tasks following installation -... # Any other files are treated as package contents. +### Packages +A Cram package consists of a directory containing a `pkg.toml` file with the following format - + +```toml +[cram] +version = 1 + +[package] + # The package.require list names depended artifacts. + [[package.require]] + name = packages.d/some-other-package + + # (optional) The package.build list enumerates either + # inline scripts or script files. These are run as a + # package is 'built' before it is installed. + [[package.build]] + run = some-build-command + + # (optional) Hook script(s) which occur before installation. + [[package.pre_install]] + run = some-hook + + # (optional) Override installation scrpt(s). + # By default, everthing under the package directory + # (the `pkg.toml` excepted) treated is as a file to be + # installed and stow is emulated using symlinks. + [[package.install]] + run = some-install-command + + # (optional) Hook script(s) which after installation. + [[package.post_install]] + run = some-other-hook ``` -Cram reads a config dir with three groups of packages -- `packages.d/` contains a package that installs but probably shouldn't configure a given tool, package or group of files. - Configuration should be left to profiles. -- `profiles.d/` contains a profile; a group of related profiles and packages that should be installed together. -- `hosts.d/` contains one package for each host, and should pull in a list of profiles. -- Both profiles and hosts entries may specify their own "inline" packages as a convenience. +To take a somewhat real example from my own dotfiles - -The intent of this tool is to keep GNU Stow's intuitive model of deploying configs via symlinks, and augment it with a useful pattern for talking about "layers" / "packages" of related configs. +```shell +$ tree -a packages.d/tmux +packages.d/tmux +├── pkg.toml +└── .tmux.conf +``` -Cram installs the package `hosts.d/$(hostname)`, and `profiles.d/default` by default. +This TMUX package provides only my `.tmux.conf` file, and a stub `pkg.toml` that does nothing. +A fancier setup could use `pkg.toml` to install TMUX either as a `pre_install` task or by using a separate TMUX package and providing the config in a profile. + +### Metapackages + +Writing lots of packages gets cumbersome quickly, as does managing long lists of explicit dependencies. +To try and manage this, Cram provides metapackages - packages which contain no stowable files, but instad contain subpackages. + +To take a somewhat real example from my own dotfiles - + +```shell +$ tree -a -L 1 profiles.d/macos +profiles.d/macos +├── pkg.toml +├── emacs/ +├── homebrew/ +└── zsh/ +``` + +The `profiles.d/macos` package depends AUTOMATICALLY on the contents of the `profiles.d/macos/emacs`, `profiles.d/macos/homebrew` and `profiles.d/macos/zsh` packages, which are normal packages. +These sub-packages can have normal dependencies on other packages both within and without the profile and install files or run scripts. + +Profiles allow users to write groups of related packages, especially configs, which go together and allows for scoped reuse of meaningful names. + +Likewise the `hosts.d/` tree allows users to store host-specific packages. ## Usage ``` -$ cram apply [--dry-run|--execute] [--optimize] +$ cram apply [--dry-run|--execute] [--optimize] [--require ] ``` The `apply` task applies a configuration to a destination directory. -The most common uses of this would be `--dry-run`, which functions as a `diff` or `--execute ~/conf ~/` for emulating Stow and installing dotfiles. +The most common uses of this would be `--dry-run` (the default), which functions as a `diff` or `--execute ~/conf ~/` for emulating Stow and installing dotfiles. + +By default `cram` installs two packages - `profiles.d/default` and `hosts.d/$(hostname -s)`. +This default can be overriden by providing `--require ` one or more times to enumerate specific packages to install. Cram always reads the `.cram.log` state file and diffs the current state against the configured state. Files and directories no longer defined by the configured state are cleaned up automatically. ``` -$ cram show +$ cram state ``` -The `list` task loads up and prints the `.cram.log` state file generated by any previous `cram apply --execute` so you can read a manifest of what cram thinks it did. +The `state` task loads up and prints the `.cram.log` state file generated by any previous `cram apply --execute` so you can read a manifest of what cram thinks it did. +This is useful because `cram` attempts to optimize repeated executions and implement change detection using the state file. + +This cache can be busted if needed by using `apply --execute --no-optimize`, which will cause cram to take all actions it deems presently required. +This can result in dangling symlinks in the filesystem. ``` -$ cram list +$ cram list [package] ``` -The `show` task lists out all available packages (eg. packages, profiles, hosts, and subpackages) as a dependency graph. +The `list` task lists out all available packages (eg. packages, profiles, hosts, and subpackages) as a dependency graph. +When provided a specific package, the details of that package (its requirements and installation task log) will be printed. ## License -Copyright Reid 'arrdem' McKenzie, 31/10/2021. +Copyright Reid 'arrdem' McKenzie, 15/02/2022. -Published under the terms of the MIT license. -See the included `LICENSE` file for more. +Published under the terms of the Anticapitalist Software License (https://anticapitalist.software). diff --git a/projects/cram/src/python/cram/__init__.py b/projects/cram/src/python/cram/__init__.py index e69de29..8326682 100644 --- a/projects/cram/src/python/cram/__init__.py +++ b/projects/cram/src/python/cram/__init__.py @@ -0,0 +1,4 @@ +__version__ = "0.1.0" +__author__ = "Reid D. 'arrdem' McKenzie " +__copyright__ = "Copyright 2020" +__license__ = "https://anticapitalist.software/" diff --git a/projects/cram/src/python/cram/__main__.py b/projects/cram/src/python/cram/__main__.py index 9636581..ff9f0e9 100644 --- a/projects/cram/src/python/cram/__main__.py +++ b/projects/cram/src/python/cram/__main__.py @@ -8,6 +8,7 @@ import pickle import sys from typing import List +from . import __version__, __author__, __license__, __copyright__ from .v0 import PackageV0, ProfileV0 from .v1 import PackageV1, ProfileV1 @@ -148,6 +149,21 @@ def scrub(old_fs: Vfs, new_fs: Vfs) -> Vfs: @click.group() +@click.version_option(version=1, message=f"""Cram {__version__} + +Documentation + https://github.com/arrdem/source/tree/trunk/projects/cram/ + +Features + - 0.0.0 legacy config format + - 0.1.0 TOML config format + - 0.1.0 log based optimizer + - 0.1.0 idempotent default for scripts + +About + {__copyright__}, {__author__}. + Published under the terms of the {__license__} license. +""") def cli(): pass @@ -175,7 +191,7 @@ def do_apply(confdir, destdir, state_file, execute, optimize, require, exec_idem # Middleware processing of the resulting filesystem(s) executable_fs = scrub(old_fs, new_fs) if optimize: - executable_fs = simplify(old_fs, new_fs, + executable_fs = simplify(old_fs, executable_fs, exec_idempotent=exec_idempotent) # Dump the new state. diff --git a/projects/cram/src/python/cram/common.py b/projects/cram/src/python/cram/common.py index 3c9df2b..eacfbe7 100644 --- a/projects/cram/src/python/cram/common.py +++ b/projects/cram/src/python/cram/common.py @@ -10,6 +10,7 @@ from vfs import Vfs SHELL = "/bin/sh" + def sh(cmd: List[str], /, env: Optional[dict] = None): diff --git a/projects/cram/src/python/cram/v1.py b/projects/cram/src/python/cram/v1.py index ef58f66..f830dad 100644 --- a/projects/cram/src/python/cram/v1.py +++ b/projects/cram/src/python/cram/v1.py @@ -26,7 +26,7 @@ def tempf(name): class PackageV1(Package): """The v1 package format.""" - SPECIAL_FILES = ["config.toml"] + SPECIAL_FILES = ["pkg.toml"] _config = None def config(self): diff --git a/projects/cram/test.sh b/projects/cram/test.sh index d513139..974c9b5 100755 --- a/projects/cram/test.sh +++ b/projects/cram/test.sh @@ -83,3 +83,14 @@ rm -r "${dest}"/* # If multiple packages provide the same _effective_ script, do it once ./cram apply --require packages.d/p6 --require packages.d/p7 --execute test/ "${dest}" | sort | uniq -c | grep "/tmp/stow/b5bea41b6c623f7c09f1bf24dcae58ebab3c0cdd90ad966bc43a45b44867e12b.sh" | grep "1 - exec" + +# Test log-based cleanup +./cram apply --require packages.d/p1 --require packages.d/p2 --execute test/ "${dest}" +[ -L "${dest}"/foo ] +[ -L "${dest}"/bar ] +# And how bar shouldn't be installed... +./cram state test/ +./cram apply --require packages.d/p1 --execute test/ "${dest}" +./cram state test/ +[ -L "${dest}"/foo ] +[ ! -L "${dest}"/bar ] diff --git a/projects/cram/test/hosts.d/test/config.toml b/projects/cram/test/hosts.d/test/pkg.toml similarity index 100% rename from projects/cram/test/hosts.d/test/config.toml rename to projects/cram/test/hosts.d/test/pkg.toml diff --git a/projects/cram/test/packages.d/p4/config.toml b/projects/cram/test/packages.d/p4/pkg.toml similarity index 100% rename from projects/cram/test/packages.d/p4/config.toml rename to projects/cram/test/packages.d/p4/pkg.toml diff --git a/projects/cram/test/packages.d/p6/config.toml b/projects/cram/test/packages.d/p6/pkg.toml similarity index 100% rename from projects/cram/test/packages.d/p6/config.toml rename to projects/cram/test/packages.d/p6/pkg.toml diff --git a/projects/cram/test/packages.d/p7/config.toml b/projects/cram/test/packages.d/p7/pkg.toml similarity index 100% rename from projects/cram/test/packages.d/p7/config.toml rename to projects/cram/test/packages.d/p7/pkg.toml diff --git a/projects/cram/test/profiles.d/default/config.toml b/projects/cram/test/profiles.d/default/pkg.toml similarity index 100% rename from projects/cram/test/profiles.d/default/config.toml rename to projects/cram/test/profiles.d/default/pkg.toml diff --git a/projects/cram/test/profiles.d/default/subpackage/config.toml b/projects/cram/test/profiles.d/default/subpackage/pkg.toml similarity index 100% rename from projects/cram/test/profiles.d/default/subpackage/config.toml rename to projects/cram/test/profiles.d/default/subpackage/pkg.toml