ubicloud/DEVELOPERS.md

Clover

Clover is the codename for Ubicloud's software. It includes a control plane, a data plane, and a web console for managing virtual machines and other applications.

It is a Ruby program that connects to Postgres.

The source code is organized based on the Roda-Sequel Stack, though several development choices have been modified. As the name indicates, this project uses Roda (for HTTP handling) and Sequel (for database queries).

Web authentication is managed with Rodauth.

Clover communicates with servers using SSH via the net-ssh library.

Tests are written using RSpec.

Code is automatically linted and formatted with RuboCop.

The web console is designed with Tailwind CSS, based on components from Tailwind UI, and uses jQuery for interactivity.

Development Environment

We recommend using mise to manage software versions. mise reads the .tool-versions file maintained in the repository.

For Ruby, obtaining a matching version is especially important because it is constrained in the Gemfile.

Install mise

If you are using mise, follow the instructions in the Getting Started Manual. There is a stand-alone installer, but you may prefer the Homebrew (brew install mise) or Debian/Ubuntu apt repository options, which are also documented on that page.

After installing mise, typing mise will display help text:

$ mise
The front-end to your dev env

Usage: mise [OPTIONS] [TASK] [COMMAND]

Commands:
[...]

mise has shell integration instructions in its manual, but included here are some short shell scripts to guide you through installing it in a conventional way.

The first task is to integrate mise with your shell. The general idea is to run mise activate $shell | source in your shell initialization file.

You can start a portable shell with sh and paste the following to automatically find the correct file:

#!/bin/sh

shell=$(basename "$SHELL")
case "$shell" in
  bash) f="$HOME/.bashrc"; [ "$(uname)" = "Darwin" ] && [ -f "$HOME/.bash_profile" ] && [ ! -f "$f" ] && f="$HOME/.bash_profile";;
  zsh)  f="$HOME/.zshrc";;
  fish) f="$HOME/.config/fish/config.fish";;
  *)    echo "Unsupported shell: $shell" >&2; exit 1;;
esac

line="mise activate $shell | source"
mkdir -p "$(dirname "$f")"; touch "$f"
grep -qF "$line" "$f" 2>/dev/null || printf "\n%s\n" "$line" >> "$f"

Activating in the shell is enough to proceed. You will need to restart your shell to apply the changes. After doing so, running mise doctor should report no problems.

For additional convenience, you can optionally install mise autocompletion. The idea is to run mise completion in the appropriate completion directory. This is straightforward for bash and fish:

#!/bin/sh

shell=$(basename "$SHELL")
case "$shell" in
  bash) comp="$HOME/.bash_completion.d/mise";;
  fish) comp="$HOME/.config/fish/completions/mise.fish";;
  *)    echo "Only bash and fish are supported by this script." >&2; exit 1;;
esac

mkdir -p "$(dirname "$comp")"
mise completion "$shell" > "$comp"
echo "Installed mise completions to $comp"

zsh is more challenging because it has no default completion path in $HOME. The script below sets up a conventional completion directory in $HOME:

#!/bin/sh

compdir="$HOME/.local/share/zsh/site-functions"
compfile="$compdir/_mise"

mkdir -p "$compdir"
mise completion zsh > "$compfile"

# Add fpath and compinit to .zshrc if not present
zshrc="$HOME/.zshrc"
grep -qF "$compdir" "$zshrc" 2>/dev/null || \
  printf '\nfpath=(%s $fpath)\n' "$compdir" >> "$zshrc"
grep -qF "compinit" "$zshrc" 2>/dev/null || \
  printf '\nautoload -U compinit; compinit\n' >> "$zshrc"

echo "Installed mise zsh completion to $compfile and enabled it in $zshrc"

Decide How to Get Postgres

People have more opinions about how to manage their Postgres version (e.g., Postgres.app, brew, apt install, etc.), and exact version matching is less important. If you don't have a preference, we suggest using mise to manage Postgres.

Managing Postgres with mise will increase the number of system dependencies you need to install to compile it. Instructions on what to install are provided in the next section.

If you choose to use mise to compile and install Postgres, you can run:

ln -s mise.local.toml.template mise.local.toml

mise.local.toml is a file that mise reads and is not committed to the source. mise.local.toml.template is committed and updated occasionally for new Postgres versions, though mise does not read it.

Installing System Dependencies

mise will compile Ruby and/or Postgres. Additionally, some Ruby gems require compilation. For all of this, you must have a C compiler, a Rust compiler, and various libraries. There is documentation listing the commands you can use for each platform (e.g., Homebrew on macOS, or Ubuntu).

You will also need dependencies installed on your system to compile Postgres.

For quick reference, here are some recipes for the most common platforms we use.

Homebrew:

xcode-select --install

# Ruby
brew install openssl@3 readline libyaml gmp autoconf

# Postgres
brew install gcc readline zlib curl ossp-uuid icu4c pkg-config

Debian/Ubuntu based:

# Ruby
apt-get install autoconf patch build-essential rustc libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libgmp-dev libncurses5-dev libffi-dev libgdbm6 libgdbm-dev libdb-dev uuid-dev

# Postgres
apt-get install build-essential libssl-dev libreadline-dev zlib1g-dev libcurl4-openssl-dev uuid-dev icu-devtools libicu-dev

mise install

Finally, after installing mise, activating it in your shell, and installing system dependencies, run:

mise install

This will install all required dependencies. You can then verify that these dependencies are active:

$ which ruby
/home/youruser/.local/share/mise/installs/ruby/3.2.8/bin/ruby
$ which postgres
/home/youruser/.local/share/mise/installs/postgres/15.8/bin/postgres
$ which node
/home/youruser/.local/share/mise/installs/node/23.6.0/bin/node
$ which go
/home/youruser/.local/share/mise/installs/go/1.24.0/bin/go

Checking mise-set Environment Variables

Mise exports additional environment variables besides $PATH, and some of them are useful to know. You can see them in shell format with mise env:

$ mise env
set -gx GOBIN /home/youruser/.local/share/mise/installs/go/1.24.0/bin
set -gx GOROOT /home/youruser/.local/share/mise/installs/go/1.24.0
set -gx LD_LIBRARY_PATH /home/youruser/.local/share/mise/installs/postgres/15.8/lib
set -gx PATH '/home/youruser/.local/share/mise/installs/ruby/3.2.8/bin:/home/youruser/.local/share/mise/installs/postgres/15.8/bin:/home/youruser/.local/share/mise/installs/node/23.6.0/bin:/home/youruser/.local/share/mise/installs/go/1.24.0/bin:/home/youruser/.local/share/mise/installs/direnv/2.35.0:/home/youruser/.local/share/mise/installs/yq/4.44.2:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin'
set -gx PGDATA /home/youruser/.local/share/mise/installs/postgres/15.8/data

After direnv setup you need to source your shell's startup files or start a new shell.

Now, upon cd-ing into the clover directory -- which already has a .envrc committed -- you should see something like:

direnv: error /home/ubicloud/code/clover/.envrc is blocked. Run `direnv allow` to approve its content

Okay, let's allow direnv in this directory:

$ asdf exec direnv allow .
direnv: loading ~/code/clover/.envrc
direnv: using asdf
direnv: Creating env file /home/ubicloud/.cache/asdf-direnv/env/2363097900-478416608-3909218245-3753665172
[...]

You should still be able to get the same Ruby version:

$ ruby --version
ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]

But, instead of being resolved through a "shim" program, the binary is referenced directly:

$ which ruby
/home/ubicloud/.asdf/installs/ruby/3.2.5/bin/ruby

Installing Postgres

You will need dependencies to compile Postgres installed on your system.

First, set some autoconf ./configure options to be passed to Postgres:

$ echo "POSTGRES_EXTRA_CONFIGURE_OPTIONS='--with-uuid=e2fs --with-openssl'" > ~/.asdf-postgres-configure-options

Then run:

$ asdf install postgres

There are many alternative ways to get Postgres, e.g. via system package manager, Homebrew, Postgres.app, etc. They are all acceptable, our version requirements for Postgres are more relaxed than with Ruby.

Installing Node.js

Node.js is required to work with frontend tooling such as tailwind-cli.

Install Node.js. asdf will consult the .tool-versions file to select the version.

$ asdf install nodejs

Having done this, you can see if your $PATH finds the node "shim" generated by asdf and consult the version:

$ which node
/home/ubicloud/.asdf/installs/nodejs/22.9.0/bin/node
$ node --version
v22.9.0

Finishing up with asdf

You may need to re-generate your direnv cache when adding programs that satisfy version requirements after having generated the direnv directory cache. This can be done via touch .envrc:

$ touch .envrc
direnv: loading ~/code/clover/.envrc
direnv: using asdf
direnv: Creating env file /home/ubicloud/.cache/asdf-direnv/env/2363097900-478416608-196231759-3753665172
direnv: loading ~/.cache/asdf-direnv/env/2363097900-478416608-196231759-3753665172
direnv: using asdf direnv 2.34.0
direnv: using asdf nodejs 22.9.0
direnv: using asdf postgres 15.8
direnv: loading ~/.asdf/plugins/postgres/bin/exec-env
direnv: using asdf ruby 3.2.5
direnv: loading ~/.asdf/plugins/ruby/bin/exec-env
direnv: export +LD_LIBRARY_PATH +PGDATA +RUBYLIB ~PATH

Although reading this output closely is seldom necessary, here we can see all the paths exported by plugins activated by .tool-versions. Note that $PGDATA is exported. Thus, we can start Postgres:

$ postgres -D $PGDATA
2024-10-15 13:22:43.682 PST [36002] LOG:  starting PostgreSQL 15.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4), 64-bit

Also note LD_LIBRARY_PATH:

$ printenv LD_LIBRARY_PATH
/home/ubicloud/.asdf/installs/postgres/15.8/lib

This is important to be able to compile client drivers against the correct libpq.

Setting up Databases

Clover uses one database per installation, but is developed using two such installations, and thus, two databases. One environment is called "development" and the other "test", and they each have a database: clover_development and clover_test. Only one user is used to connect to both databases, though, named clover.

Presuming you have set up Postgres using mise, run a server in a dedicated terminal window with postgres -D $PGDATA set aside, and then create the user and databases:

$ createuser -U postgres clover
$ createuser -U postgres clover_password
$ rake setup_database\[development,false\]
$ rake setup_database\[test,false\]

The clover_test database is used by automated tests, and is prone to automatic truncation and the like. clover_development is the database used by default, where the developer (you) manages the data.

For example, you might create records in clover_development addressing a few hosts you bought on Hetzner and then experiment with creating and destroying VMs this way. Looking at the clover_test database is rare, usually when working on or debugging the testing infrastructure itself.

Running migrations

The setup_database task will drop databases if they exist, create databases and then migrate them.

If you have already setup the databases, and you want to run new migrations to update them to the latest schema:

$ rake test_up
$ rake dev_up

The rake task sets RACK_ENV and the .env.rb generated by overwrite_envrb interprets this to find the right configuration, including the database name to migrate.

Configuration

You can read config.rb to see what environment variables are used.

CLOVER_DATABASE_URL and RACK_ENV are mandatory, but for running tests, you will also need to set CLOVER_SESSION_SECRET and CLOVER_COLUMN_ENCRYPTION_KEY. The former is necessary for web (but not database model) tests, the latter is necessary for any test that uses an encrypted column.

Our programs load a file .env.rb if present to run arbitrary Ruby code to set up the environment. You can generate a sensible .env.rb with rake overwrite_envrb:

$ rake overwrite_envrb
$ cat .env.rb
case ENV["RACK_ENV"] ||= "development"
when "test"
  ENV["CLOVER_SESSION_SECRET"] ||= "mbvxopHlcCTWxT6E62weAT+9vxAr1BJp7X3OuQ4K+fFYOLwM20wBVHLuM5tITJDZcEMy2luUD9CDbfgU9okiCw=="
  ENV["CLOVER_DATABASE_URL"] ||= "postgres:///clover_test?user=clover"
  ENV["CLOVER_COLUMN_ENCRYPTION_KEY"] ||= "EWLXd9OzR7Rvs254gVOE9BeTv3fBoZeysOjcNReu5zw="
else
  ENV["CLOVER_SESSION_SECRET"] ||= "/UBMRpwQ5NN3NmSM81FtqDfaaRWhqxbmfFXMxMA2fjcdUk53SZF5n4SKd+uAIpPgPWx1ItRGq/JW1yzQqx0PdQ=="
  ENV["CLOVER_DATABASE_URL"] ||= "postgres:///clover_development?user=clover"
  ENV["CLOVER_COLUMN_ENCRYPTION_KEY"] ||= "9sljUbAiMmH0uiYE6lM64Tix72ehGr0W7yFrbpD+l4s="
end

Here we can see that .env.rb chooses the database and keys in question based on RACK_ENV, defaulting to development.

Note that these keys change with every execution of overwrite_envrb, so generating a new .env.rb can result in encrypted data in your clover_development database being indecipherable. You are unlikely to generate this file often, and can probably use the same .env.rb with minor modifications for years.

Installing Ruby Dependencies a.k.a. Gems

Like most programming environments, Ruby has an application-level dependency management system, called RubyGems. We manage those versions through the program bundler, which itself we get through the low-level gem command:

$ which gem
/home/youruser/.local/share/mise/installs/ruby/3.2.8/bin/gem
$ gem install bundler
Fetching bundler-2.6.7.gem
[...]
$ bundle install
Bundle complete! 63 Gemfile dependencies, 178 gems now installed.
[...]

Bundler's function is to solve complex gem version constraint upgrades (when running bundle update) and to generate and interpret Gemfile.lock to select the correct Gem versions to be loaded when multiple versions are installed. This is done via bundle exec or loading bundler in application code, such as loader.rb's call to Bundler.setup. In general, bundle exec is necessary when Clover does not control the entry point into the program, such as rubocop (to lint code) or rspec (to run tests):

$ bundle exec rubocop

But it's not necessary with programs in bin that we control and load loader.rb right away, as a convenience:

$ ./bin/pry

It's harmless yet duplicative to run:

$ bundle exec bin/pry

Formatting and Linting code with RuboCop

RuboCop is a code linter and rewriter. It can take care of all minor formatting issues automatically, e.g. indentation. You can run auto-correction with bundle exec rubocop -a

If you ran overwrite_envrb, it generates a file that's prone to correction by RuboCop:

$ bundle exec rubocop -a
Inspecting 68 files
C...................................................................

Offenses:

.env.rb:6:34: C: [Corrected] Style/StringLiterals: Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
    ENV["CLOVER_DATABASE_URL"] ||= 'postgres:///clover_test?user=clover'
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

68 files inspected, 1 offense detected, 1 offense corrected

Some useful corrections are only made with bundle exec rubocop -A (upper case A) which applies "unsafe" corrections that may alter the semantics of the program.

Running the tests

With the database running and the test database up to date with migrations, you can run the tests:

$ bundle exec rspec

or even just:

$ rake

As the default rake task runs all the tests.

You can collect coverage by setting:

$ COVERAGE=1 rake

You can run a specific file or line when using bundle exec rspec:

$ bundle exec rspec ./spec/model/strand_spec.rb
$ bundle exec rspec ./spec/model/strand_spec.rb:10

There is editor integration for RSpec that are very useful. rspec-mode for emacs (as seen in M-x list-packages) has lisp procedures rspec-verify to run rspec on the file where the point is, rspec-verify-single to run it on the line the point is at, and rspec-rerun to run rspec the same way as whatever came last, which is excellent when editing code that should affect the outcome of a test. There is also rspec-verify-all which runs all the specs, but this is less essential than running one or a few specs with editor integration.

Assuredly, there is all this and more in other editor environments.

Running Web Console

Web Console is designed with Tailwind CSS. Tailwind CSS works by scanning all of your HTML files, JavaScript components, and any other templates for class names, generating the corresponding styles and then writing them to a static CSS file. You need to generate CSS file before running web console if you do not want to see HTML files without any style.

We manage node module versions through npm. It's installed with nodejs package.

$ which npm
/home/youruser/.local/share/mise/installs/node/23.6.0/bin/npm
$ npm install
[...]
added 46 packages, removed 19 packages, changed 41 packages, and audited 527 packages in 1s

14 packages are looking for funding
    run `npm fund` for details

found 0 vulnerabilities

Now we can build CSS file. If you do development on UI, you can run npm run watch on separate terminal window to see changes realtime.

$ npm run prod
> prod
> npx tailwindcss -o assets/css/app.css --minify

Rebuilding...

Done in 767ms.

assets/css/app.css should be updated.

After that, start up the web server.

$ bundle exec rackup

And then visiting http://localhost:9292, you can create an account. Check the rackup log for the verification link to navigate to, in production, we would send that output as email. Having verified, log in. You'll see the "Getting Started" page.

When you change any template file, format them with erb-formatter:

$ rake linter:erb_formatter

Metrics setup

To have the metrics system function during local development, start a VictoriaMetrics instance in the background with:

$ victoria-metrics -storageDataPath var/victoria-metrics-data

Cloudifying a Host for Development

We show cloudifying a host from Hetzner, but the principles should work everywhere. Make sure that the Hetzner instance has at least one One additional subnet /29 ordered and Ubuntu 24.04 LTS base is installed.

1. Set the environment variables in .env.rb;

   ENV["HETZNER_USER"] ||= HETZNER_ACCOUNT_ID
   ENV["HETZNER_PASSWORD"] ||= HETZNER_ACCOUNT_PASS
   ENV["HETZNER_SSH_PUBLIC_KEY"] ||= YOUR_PUBLIC_SSH_KEY
   ENV["HETZNER_SSH_PRIVATE_KEY"] ||= YOUR_PRIVATE_SSH_KEY
   ENV["OPERATOR_SSH_PUBLIC_KEYS"] ||= YOUR_PUBLIC_SSH_KEY\nOTHER_PUBLIC_SSH_KEYS
   

2. In terminal 1, start the respirate process:

   $ ./bin/respirate
   

3. In terminal 2, connect to REPL console running ./bin/pry and start cloudification:

   VM_HOST_IP = ""
   VM_HOST_ID = ""
   default_boot_images = ["ubuntu-noble", "ubuntu-jammy", "debian-12", "almalinux-9"]
   
   st = Prog::Vm::HostNexus.assemble(VM_HOST_IP, provider_name: "hetzner", location_id: Location::HETZNER_FSN1_ID, server_identifier: VM_HOST_ID, default_boot_images: default_boot_images)
   vmh = st.subject
   

4. Get back to terminal 2 and observe VmHost cloudification process

   while true
     lbl = vmh.strand.reload.label
     puts lbl
     break if lbl == "wait"
     sleep 2
   end
   

When the strand responsible for the VmHost goes to wait state, this means the host is ready to be used for Ubicloud services. Now you can use the web console to create resources, such as VMs.

Conclusion

That's everything there is to know. As exercise, you can consider inserting a crash into some source under test (e.g. strand.rb) and try to make the tests fail with a backtrace:

An edited strand.rb:

[...]
def self.lease(id)
  fail "my first crash"
  affected = DB[<<SQL, id].first
[...]

And, the crash:

$ bundle exec rspec ./spec/model/strand_spec.rb

Randomized with seed 60335

Strand
    can load a prog
    can run a label (FAILED - 1)
    can take leases (FAILED - 2)

Failures:

    1) Strand can run a label
        Failure/Error: st.run

        RuntimeError:
        my first crash
        # ./model/strand.rb:15:in `lease'
        # ./model/strand.rb:9:in `lease'
        # ./model/strand.rb:44:in `run'
        # ./spec/model/strand_spec.rb:24:in `block (2 levels) in <top (required)>'
        # ./spec/spec_helper.rb:41:in `block (3 levels) in <top (required)>'
        # ./spec/spec_helper.rb:40:in `block (2 levels) in <top (required)>'