Frappix

A Frappe Development & Deployment Environment

_{It is best for now, to join the community in the chat until the docs are more elaborate!}

Frappix is a development and deployment environment designed to cover the full software delivery lifecycle from development to deployment and operation for Frappe-based projects.

It is intended for developers and operator alike in their respective role in customer-facing or educational projects.

It can be used in simple scenarios to prototype apps that could later be run on Frappé Cloud but also for very complex production deployments which required extensive customization and fork-like patching of the upstream framework.

Motivation

Frappix bridges the gap between system dependencies and such that are already available in Python.

It brings the power of the entire software ecosystem to Frappé, not only the Python ecosystem.

It leverages Nix to achieve (close to) reproducible builds of your deployment artifacts, while Nixpkgs is leveraged for its vast amount of up to date and readily available packages across various language ecosystems.

For example, it is near trivial to set up and run a LLM efficiently via llama.cpp alongside your production setup, while it is trivial to provide a swalwart email service in-scope on the same project, set up a nightly backup with StorJ, tweak your database performance with hugepages, host you plausible analytics instance alongside, or even spin up an entire private chat solution based on the Matrix protocol, etc.

Battle tested

Frappix, and it's predecessor, has served the author very well during the last year in a complex and highly sofisticated production environment.

Please contact me via the above Matrix Chat or the Frappé Forum if you have any further inquiry.

Installation

To use frappix, you can install all its decpencencies with:

$SHELL <(curl -L https://blaggacao.github.io/frappix/install)

You can inspect the bill of material of this install script in its source.

This will install, the following components on which Frappix is built:

Nix: global package manager & language interpreter
Direnv: tool to manage environments per folder
Nom: nix output monitfor for better display
Frappix Tool: runs repository tasks

Use Templates

To use one of the available project templates, run:

PROJECT=<my-project>
TEMPLATE=<my-chosen-template>
nix flake new "$PROJECT" -t "github:blaggacao/frappix#$TEMPLATE"

Simple Frappé

To start your project with a simple Frappé setup, run:

PROJECT=<my-project>
TEMPLATE=frappe
nix flake new "$PROJECT" -t "github:blaggacao/frappix#$TEMPLATE"

Finish Setup

When you change into the newly created directory, direnv will ask you to approve the environment hook.
- Don't do so, yet!
First, initialize a git repository in this folder: git init -b main
Then, lock environment dependencies with: git add . && nix flake lock
Next, add and commit your new files with: git add . && git commit -m "Initial commit"
Now, accept the environment file with: direnv allow

Enable Extra Repository Tooling

The extra tooling provides:

Formatter support
Commit lint support
Documentation support
Editorconfig template

To enable it, change the following value in tools/shells.nix:

{
-   bench.enableExtraProjectTools = false;
+   bench.enableExtraProjectTools = true;
}

Run `bench new-app`

This will walk you through the creation of a new app.

Note: this is an upstream bench command and not yet seamlessly integrated with Frappix.

Add the production pin for your new sources

Note: this is dumped from memory after the fact, expect imprecision.

Go to apps/_pins/config.toml
Declare your new source; take examples from the Frappix repository
Run nvchecker -c config.toml to pin the sources and generate the files

Note: you may have seen <app>.passthru arguments in the Frappix repository's config.toml

since declares how far the shallow clone should go back (acquiring a common ancestor if since isn't in the parent tree of the current branch)
upstream declares a fetch configuration to only fetch the portions of history from upstream that you may be interested in

These setups are done on first clone, so they don't have any effect on your newly created app, since you already initialized the repository locally.

Add `./apps/<my-app>.nix`

Change the name of the file to the name of your app and paste the following content and go through the comments:

{
  # access to the pinned frappix sources
  appSources,
  # helper function to extract metadata from frappe apps
  extractFrappeMeta,
  # access to a nixpkgs library
  lib,
  # python builder instrumentation
  buildPythonPackage,
  pythonRelaxDepsHook,
  flit-core,
  python,
}:
buildPythonPackage rec {
  inherit
    (extractFrappeMeta src)
    pname
    version
    format
    ;

  # change to access your app's sources
  inherit (appSources.my-app) src;
  # change for a rudimentary `import my-app`-like assertion
  # of the final package as if run from a python repl
  pythonImportsCheck = ["my-app"];

  nativeBuildInputs = [
    # this tool helps to relax dependencies in case the prepackages libraries
    # are of a different version; prepackages libraries have the huge benefit
    # of being readily available and cached and most of the time work just fine
    pythonRelaxDepsHook
    # the upstream app template uses flit-core these days
    # for older packages, you best update the build system and upstream your patch
    flit-core
  ];

  # additional python or other dependencies
  # for all available packages, see: https://search.nixos.org/packages
  propagatedBuildInputs = with python.pkgs; [
    # rembg
  ];

  # typically, we want to simply relax all dependency versions and use the prepackaged ones;
  # if a version does _really_ not work, you'll need to package the correct python package
  # yourself; for that: get help in the Matrix Chat!
  pythonRelaxDeps = true;
}

Because Nix is designed to only load files which are principally under version controll, you'll at least to git add ./apps/\.nix before it will be visible to the builder.

Add it to `./apps/pkgs.nix`

Add your new package to ./apps/pkgs.nix, change and uncomment the part about the custom app.

  inject = _: prev: {
    # extend the frappix package set
    frappix = prev.frappix.overrideScope (finalFrappix: prevFrappix: {
      # inject your pinned sources (if any) into the frappix build pipeline
      appSources = prevFrappix.appSources.overrideScope (_: _: _pins);
      # add custom apps that are not yet packaged by frappix
      # my-app = finalFrappix.callPackage ./my-app.nix {};
    });
  };

pkgs.frappix is now populated with your new app.

It is now available for ubiquitous use under that handle in deployment artifacts, development environments, etc.

Add it to `./tools/shells.nix`

Chose the right name from the previous step and uncomment where it reads:

      bench.apps = with pkgs.frappix; [
        # my-app
      ];

Finally `devenv reload`

This ensures that apps.txt will be updated.

TODO: make this part of automatism.

Configure `fjsd`

To obtain semantic diff on Frappé JSON, within the git repository of your new app, run:

git config --local --add diff.fsjd.command "fsjd --git"
cat << CONFIG >> .git/info/attributes
*.json diff=fsjd
CONFIG

Custom Upstream (Frappé / ERP Next)

There may be many situations where you need to patch upstream Frappé / ERP Next, either temporary or permanently.

For example, it may take time to upstream a patch, and you need to carry your fixes in the meantime.

You can easily pin your version of sources with the shipped source pinning mechanism.

The sources (_pins) will be injected via apps/pkgs.nix on this line into the build and deployment pipeline:

{
  appSources = prevFrappix.appSources.overrideScope (_: _: _pins);
}

Add custom dependencies

If you also need to add custom dependencies, it will be only slightly more difficult.

You need to ensure that the dependencies are packaged in your package set.

You can check if they are contained upstream in your current Nixpkgs pin
Or you can package them yourself in an overlay; for this you'd add something like the following snippet to the inject function in apps/pkgs.nix:

{
  inject = final: prev {
    pythonPackagesExtensions =
      prev.pythonPackagesExtensions
      ++ [
        (pyFinal: pyPrev: {
          python-qrcode = pyFinal.callPackage ./python-qrcode.nix {};
          whatsfly = pyFinal.callPackage ./whatsfly.nix {};
          matrix-nio = pyFinal.callPackage ./matrix-nio.nix {};
          vrp-cli = pyFinal.callPackage ./vrp-cli {};
        })
      ];
    # [...]
  };
}

You can add them into the upstream build instructions like so, within the inject function:

{
  frappix = prev.frappix.overrideScope (finalFrappix: prevFrappix: {
    frappe = prevFrappix.frappe.overridePythonAttrs (o: {
      propagatedBuildInputs = with prevFrappix.frappe.pythonModule.pkgs;
        o.propagatedBuildInputs
        ++ [
          matrix-nio
          authlib
          whatsfly
        ];
    });
    erpnext = prevFrappix.erpnext.overridePythonAttrs (o: {
      propagatedBuildInputs = with prevFrappix.erpnext.pythonModule.pkgs;
        o.propagatedBuildInputs
        ++ [
          vrp-cli
          shapely
          pyproj
          numpy
          scipy
          geojson
        ];
    });

  };
}

Run tests

Use the test rig module
Build the VM
Run it
Run tests

Example of using the test rig module:

let
  # [...]
  inherit (inputs.frappix.nixosModules) testrig frappix;
in rec {
  test-HOST = {
    config,
    lib,
    ...
  }: {
    imports = [
      HOST
      testrig
    ];
    # maybe some manual adjustments and override necessary for the test
  };
  HOST = {
    # your production config
  };
}

Build the VM with:

TODO: incorporate into frx more elegantly

nix build .\#nixosConfigurations.deploy-test-HOST.config.system.build.vm

Run the VM in headless mode:

sudo ensures we can bind to the low ports 80 & 443 to fully test the VM

# launch the VM
QEMU_NET_OPTS="hostfwd=tcp:127.0.0.1:2222-:22" sudo ./result/bin/run-HOST-vm; reset

TODO: Run tests:

Just run ...

bench ...

Frappix Documentation