ubicloud/DEVELOPERS.md

# Clover

Clover is the codename for Ubicloud's software, which consists of the control plane, data plane, and web console program for managing virtual machines and other applications.

It's a Ruby program that connects to Postgres.

The source code organization is based on the [Roda-Sequel
Stack](https://github.com/jeremyevans/roda-sequel-stack), though a
number of development choices have been modified.  As the name
indicates, this project uses [Roda](https://roda.jeremyevans.net/)
(for HTTP code) and [Sequel](http://sequel.jeremyevans.net/) (for
database queries).

Web authentication is managed with
[Rodauth](http://rodauth.jeremyevans.net/).

It communicates with servers using SSH, via the library
[net-ssh](https://github.com/net-ssh/net-ssh).

The tests are written using [RSpec](https://rspec.info/).

Code is automatically linted and formatted using
[RuboCop](https://rubocop.org/).

Web console is designed with [Tailwind CSS](https://tailwindcss.com)
based on components from [Tailwind UI](https://tailwindui.com). It uses
jQuery for interactivity.

## Development Environment

We suggest using [asdf-vm](https://github.com/asdf-vm/asdf) to manage
software versions.  There is a [.tool-versions file](.tool-versions)
that `asdf-vm` reads, and it is kept up to date.

In the case of Ruby, obtaining a matching version is most
important, because it is constrained in the [Gemfile](Gemfile).

Though, any method of obtaining Ruby and Postgres is adequate.

### Install asdf-vm and plugins

If using `asdf-vm`, follow the instructions at the [Getting Started
Manual](https://asdf-vm.com/guide/getting-started.html). There are
three general steps:

1. Download some common dependencies, `git` and `curl`. You may
   already have them.
2. Use `git` to clone `asdf-vm` into your home directory
3. Source it into your shell automatically

Having done so, typing `asdf` will yield a bunch of help text:

```sh
$ asdf
version: v0.14.1-f00f759

MANAGE PLUGINS
asdf plugin add <name> [<git-url>]      Add a plugin from the plugin repo OR,
[...]
```

We like to have these plugins (you can paste these commands):

```sh
$ asdf plugin add ruby
$ asdf plugin add direnv
$ asdf plugin add postgres
$ asdf plugin add nodejs
$ asdf plugin add golang
```

Once you have the plugins, you can start to install the software the
plugin supports.  Let's first install Ruby.

### Installing Ruby

First, install some system dependencies, such as a C and Rust compiler.
[There is documentation listing the commands you can use for each
platform](https://github.com/rbenv/ruby-build/wiki#suggested-build-environment)
(e.g. Macintosh Homebrew, or Ubuntu).

After that, install Ruby.  `asdf` will consult the `.tool-versions`
file to select the version.

```sh
$ asdf install ruby
```

Having done this, you can see if your `$PATH` finds the `ruby` "shim"
generated by `asdf` and consult the version:

```sh
$ which ruby
/home/ubicloud/.asdf/shims/ruby
$ ruby --version
ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
```

### Installing asdf-direnv

We find use of `asdf` with
[asdf-direnv](https://github.com/asdf-community/asdf-direnv) almost
obligatory, for the reasons discussed in its README.  Let's set it up
as a user-global tool, and not a project-local one:

```sh
$ asdf direnv setup --version latest
$ echo "direnv $(direnv --version)" >> ~/.tool-versions
```

After `direnv setup` you need to source your shell's startup files or
start a new shell.

Now, upon `cd`-ing into the clover directory -- which already has a
[.envrc](.envrc) committed -- you should see something like:

```sh
direnv: error /home/ubicloud/code/clover/.envrc is blocked. Run `direnv allow` to approve its content
```

Okay, let's allow direnv in this directory:

```sh
$ asdf exec direnv allow .
direnv: loading ~/code/clover/.envrc
direnv: using asdf
direnv: Creating env file /home/ubicloud/.cache/asdf-direnv/env/2363097900-478416608-3909218245-3753665172
[...]
```

You should still be able to get the same Ruby version:

```sh
$ ruby --version
ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
```

But, instead of being resolved through a "shim" program, the binary is
referenced directly:

```sh
$ which ruby
/home/ubicloud/.asdf/installs/ruby/3.2.5/bin/ruby
```

### Installing Postgres

You will need
[dependencies](https://github.com/smashedtoatoms/asdf-postgres#dependencies)
to compile Postgres installed on your system.

First, set some autoconf `./configure` options to be passed to
Postgres:

```sh
$ echo "POSTGRES_EXTRA_CONFIGURE_OPTIONS='--with-uuid=e2fs --with-openssl'" > ~/.asdf-postgres-configure-options
```

Then run:

```sh
$ asdf install postgres
```

There are many alternative ways to get Postgres, e.g. via system
package manager, Homebrew, Postgres.app, etc.  They are all
acceptable, our version requirements for Postgres are more relaxed
than with Ruby.

### Installing Node.js

Node.js is required to work with frontend tooling such as tailwind-cli.

Install Node.js. `asdf` will consult the `.tool-versions`
file to select the version.

```sh
$ asdf install nodejs
```

Having done this, you can see if your `$PATH` finds the `node` "shim"
generated by `asdf` and consult the version:

```sh
$ which node
/home/ubicloud/.asdf/installs/nodejs/22.9.0/bin/node
$ node --version
v22.9.0
```

### Finishing up with asdf

You may need to re-generate your `direnv` cache when adding programs
that satisfy version requirements after having generated the direnv
directory cache.  This can be done via `touch .envrc`:

```sh
$ touch .envrc
direnv: loading ~/code/clover/.envrc
direnv: using asdf
direnv: Creating env file /home/ubicloud/.cache/asdf-direnv/env/2363097900-478416608-196231759-3753665172
direnv: loading ~/.cache/asdf-direnv/env/2363097900-478416608-196231759-3753665172
direnv: using asdf direnv 2.34.0
direnv: using asdf nodejs 22.9.0
direnv: using asdf postgres 15.4
direnv: loading ~/.asdf/plugins/postgres/bin/exec-env
direnv: using asdf ruby 3.2.5
direnv: loading ~/.asdf/plugins/ruby/bin/exec-env
direnv: export +LD_LIBRARY_PATH +PGDATA +RUBYLIB ~PATH
```

Although reading this output closely is seldom necessary, here we can
see all the paths exported by plugins activated by `.tool-versions`.
Note that `$PGDATA` is exported.  Thus, we can start Postgres:

```sh
$ postgres -D $PGDATA
2024-10-15 13:22:43.682 PST [36002] LOG:  starting PostgreSQL 15.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4), 64-bit
```

Also note `LD_LIBRARY_PATH`:

```sh
$ printenv LD_LIBRARY_PATH
/home/ubicloud/.asdf/installs/postgres/15.4/lib
```

This is important to be able to compile client drivers against the
correct `libpq`.

### Setting up Databases

Clover uses one database per installation, but is developed using two
such installations, and thus, two databases.  One environment is
called "development" and the other "test", and they each have a
database: `clover_development` and `clover_test`.  Only one user is
used to connect to both databases, though, named `clover`.

Presuming you have set up Postgres using `asdf-vm`, run a server in a
dedicated terminal window with `postgres -D $PGDATA` set aside, and
then create the user and databases:

```sh
$ createuser -U postgres clover
$ createuser -U postgres clover_password
$ rake setup_database\[development,false\]
$ rake setup_database\[test,false\]
```

The `clover_test` database is used by automated tests, and is prone to
automatic truncation and the like. `clover_development` is the
database used by default, where the developer (you) manages the data.

For example, you might create records in `clover_development`
addressing a few hosts you bought on Hetzner and then experiment with
creating and destroying VMs this way.  Looking at the `clover_test`
database is rare, usually when working on or debugging the testing
infrastructure itself.

### Running migrations

The `setup_database` task will drop databases if they exist, create databases and then migrate them.

If you have already setup the databases, and you want to run new migrations to update them
to the latest schema:

```
$ rake test_up
$ rake dev_up
```

The rake task sets `RACK_ENV` and the `.env.rb` generated by `overwrite_envrb` interprets this to find the right configuration, including the database name to migrate.

### Configuration

You can read [config.rb](config.rb) to see what environment variables
are used.

`CLOVER_DATABASE_URL` and `RACK_ENV` are mandatory, but for running
tests, you will also need to set `CLOVER_SESSION_SECRET` and
`CLOVER_COLUMN_ENCRYPTION_KEY`.  The former is necessary for web (but
not database model) tests, the latter is necessary for any test that
uses an encrypted column.

Our programs load a file `.env.rb` if present to run arbitrary Ruby
code to set up the environment.  You can generate a sensible `.env.rb`
with `rake overwrite_envrb`:

```sh
$ rake overwrite_envrb
$ cat .env.rb
case ENV["RACK_ENV"] ||= "development"
when "test"
  ENV["CLOVER_SESSION_SECRET"] ||= "mbvxopHlcCTWxT6E62weAT+9vxAr1BJp7X3OuQ4K+fFYOLwM20wBVHLuM5tITJDZcEMy2luUD9CDbfgU9okiCw=="
  ENV["CLOVER_DATABASE_URL"] ||= "postgres:///clover_test?user=clover"
  ENV["CLOVER_COLUMN_ENCRYPTION_KEY"] ||= "EWLXd9OzR7Rvs254gVOE9BeTv3fBoZeysOjcNReu5zw="
else
  ENV["CLOVER_SESSION_SECRET"] ||= "/UBMRpwQ5NN3NmSM81FtqDfaaRWhqxbmfFXMxMA2fjcdUk53SZF5n4SKd+uAIpPgPWx1ItRGq/JW1yzQqx0PdQ=="
  ENV["CLOVER_DATABASE_URL"] ||= "postgres:///clover_development?user=clover"
  ENV["CLOVER_COLUMN_ENCRYPTION_KEY"] ||= "9sljUbAiMmH0uiYE6lM64Tix72ehGr0W7yFrbpD+l4s="
end
```

Here we can see that .env.rb chooses the database and keys in question
based on `RACK_ENV`, defaulting to `development`.

Note that these keys change with every execution of `overwrite_envrb`,
so generating a new `.env.rb` can result in encrypted data in your
`clover_development` database being indecipherable.  You are unlikely
to generate this file often, and can probably use the same `.env.rb`
with minor modifications for years.

### Installing Ruby Dependencies a.k.a. Gems

Like most programming environments, Ruby has an application-level
dependency management system, called
[RubyGems](https://rubygems.org/).  We manage those versions through
the program [bundler](https://bundler.io/), which itself we get
through the low-level `gem` command:

```sh
$ which gem
/home/ubicloud/.asdf/installs/ruby/3.2.5/bin/gem
$ gem install bundler
Fetching bundler-2.5.17.gem
[...]
$ bundle install
[...]
Bundle complete! 30 Gemfile dependencies, 75 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
```

Bundler's function is to solve complex gem version constraint upgrades
(when running `bundle update`) and to generate and interpret
[Gemfile.lock](Gemfile.lock) to select the correct Gem versions to be
loaded when multiple versions are installed.  This is done via `bundle
exec` or loading bundler in application code, such as
[loader.rb](loader.rb)'s call to `Bundler.setup`.  In general, `bundle
exec` is necessary when Clover does not control the entry point into
the program, such as `rubocop` (to lint code) or `rspec` (to run
tests):

```sh
$ bundle exec rubocop
```

But it's not necessary with programs in `bin` that we control and load
`loader.rb` right away, as a convenience:

```sh
$ ./bin/pry
```

It's harmless yet duplicative to run:

```sh
$ bundle exec bin/pry
```

### Formatting and Linting code with RuboCop

RuboCop is a code linter and rewriter.  It can take care of all minor
formatting issues automatically, e.g. indentation.  You can run
auto-correction with `bundle exec rubocop -a`

If you ran `overwrite_envrb`, it generates a file that's prone to
correction by RuboCop:

```sh
$ bundle exec rubocop -a
Inspecting 68 files
C...................................................................

Offenses:

.env.rb:6:34: C: [Corrected] Style/StringLiterals: Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
    ENV["CLOVER_DATABASE_URL"] ||= 'postgres:///clover_test?user=clover'
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

68 files inspected, 1 offense detected, 1 offense corrected
```

Some useful corrections are only made with `bundle exec rubocop -A`
(upper case `A`) which applies "unsafe" corrections that may alter the
semantics of the program.

### Running the tests

With the database running and the test database up to date with
migrations, you can run the tests:

```sh
$ bundle exec rspec
```

or even just:

```sh
$ rake
```

As the default rake task runs all the tests.

You can collect coverage by setting:

```sh
$ COVERAGE=1 rake
```

You can run a specific file or line when using `bundle exec rspec`:

```sh
$ bundle exec rspec ./spec/model/strand_spec.rb
$ bundle exec rspec ./spec/model/strand_spec.rb:10
```

There is editor integration for RSpec that are very useful.
`rspec-mode` for emacs (as seen in `M-x list-packages`) has lisp
procedures `rspec-verify` to run `rspec` on the file where the point
is, `rspec-verify-single` to run it on the line the point is at, and
`rspec-rerun` to run `rspec` the same way as whatever came last, which
is excellent when editing code that should affect the outcome of a
test.  There is also `rspec-verify-all` which runs all the specs, but
this is less essential than running one or a few specs with editor
integration.

Assuredly, there is all this and more in other editor environments.

### Running Web Console

Web Console is designed with Tailwind CSS. Tailwind CSS works by scanning
all of your HTML files, JavaScript components, and any other templates
for class names, generating the corresponding styles and then writing
them to a static CSS file. You need to generate CSS file before running
web console if you do not want to see HTML files without any style.

We manage node module versions through [npm](https://www.npmjs.com). It's
installed with `nodejs` package.

```sh
$ which npm
/home/ubicloud/.asdf/installs/nodejs/22.9.0/bin/npm
$ npm install
[...]
added 86 packages, and audited 87 packages in 1s

14 packages are looking for funding
    run `npm fund` for details

found 0 vulnerabilities
```

Now we can build CSS file. If you do development on UI, you can run
`npm run watch` on separate terminal window to see changes realtime.

```sh
$ npm run prod
> prod
> npx tailwindcss -o assets/css/app.css --minify

Rebuilding...

Done in 767ms.
```

`assets/css/app.css` should be updated.

After that, start up the web server.

```sh
$ bundle exec rackup
```

And then visiting [http://localhost:9292](http://localhost:9292), you can
create an account. Check the rackup log for the verification link to navigate
to, in production, we would send that output as email. Having verified, log
in. You'll see the "Getting Started" page.

When you change any template file, format them with `erb-formatter`:

```sh
$ rake linter:erb_formatter
```

### Cloudifying a Host for Development

We show cloudifying a host from Hetzner, but the principles should work everywhere. Make sure that the Hetzner instance has at least one `One additional subnet /29` ordered and `Ubuntu 22.04 LTS base` is installed.

1. Set the environment variables in `.env.rb`;
    ```ruby
    ENV["HETZNER_USER"] ||= HETZNER_ACCOUNT_ID
    ENV["HETZNER_PASSWORD"] ||= HETZNER_ACCOUNT_PASS
    ENV["HETZNER_SSH_PUBLIC_KEY"] ||= YOUR_PUBLIC_SSH_KEY
    ENV["HETZNER_SSH_PRIVATE_KEY"] ||= YOUR_PRIVATE_SSH_KEY
    ENV["OPERATOR_SSH_PUBLIC_KEYS"] ||= YOUR_PUBLIC_SSH_KEY\nOTHER_PUBLIC_SSH_KEYS
    ```

2. In **terminal 1**, start the respirate process:
    ```sh
    $ ./bin/respirate
    ```

3. In **terminal 2**, connect to REPL console running `./bin/pry` and start cloudification:
    ```ruby
    VM_HOST_IP = ""
    VM_HOST_ID = ""
    default_boot_images = ["ubuntu-noble", "ubuntu-jammy", "debian-12", "almalinux-9"]

    st = Prog::Vm::HostNexus.assemble(VM_HOST_IP, provider_name: "hetzner", location_id: Location::HETZNER_FSN1_ID, server_identifier: VM_HOST_ID, default_boot_images: default_boot_images)
    vmh = st.subject
    ```

4. Get back to **terminal 2** and observe `VmHost` cloudification process
    ```ruby
    while true
      lbl = vmh.strand.reload.label
      puts lbl
      break if lbl == "wait"
      sleep 2
    end
    ```

When the `strand` responsible for the VmHost goes to `wait` state, this means the host is ready to be used for Ubicloud services. Now you can use the web console to create resources, such as VMs.

### Conclusion

That's everything there is to know.  As exercise, you can consider
inserting a crash into some source under test (e.g. `strand.rb`) and
try to make the tests fail with a backtrace:

An edited `strand.rb`:
```ruby
[...]
def self.lease(id)
  fail "my first crash"
  affected = DB[<<SQL, id].first
[...]
```

And, the crash:
```sh
$ bundle exec rspec ./spec/model/strand_spec.rb

Randomized with seed 60335

Strand
    can load a prog
    can run a label (FAILED - 1)
    can take leases (FAILED - 2)

Failures:

    1) Strand can run a label
        Failure/Error: st.run

        RuntimeError:
        my first crash
        # ./model/strand.rb:15:in `lease'
        # ./model/strand.rb:9:in `lease'
        # ./model/strand.rb:44:in `run'
        # ./spec/model/strand_spec.rb:24:in `block (2 levels) in <top (required)>'
        # ./spec/spec_helper.rb:41:in `block (3 levels) in <top (required)>'
        # ./spec/spec_helper.rb:40:in `block (2 levels) in <top (required)>'
```