# Calculation Engine

The Calculation Engine (CE) is the application that integrates the MUSES modules by providing an API for running scientific calculation workflows involving one or more modules. This document explains how to run the CE locally in Docker Compose and develop its code.

To run the CE in Kubernetes, take a look at [the Helm chart driving our production deployment](https://gitlab.com/nsf-muses/deployment/kubernetes/-/tree/main/charts/calculation-engine).

## Installation

1. Follow carefully the [Docker Compose installation instructions](https://docs.docker.com/compose/) for your operating system.

1. Clone the git repository from the desired release (replace `main` with a numbered release tag like `1.2.3` to checkout a specific version):

    ```bash
    # Specify the target released version
    RELEASE_TAG=1.2.3
    # Clone the git repo
    git clone -b ${RELEASE_TAG} https://gitlab.com/nsf-muses/calculation-engine.git
    # Enter the cloned repo directory for subsequent commands
    cd calculation-engine
    ```

## Configuration

### Environment

Customize the configuration as desired by setting environment variable values in a file called `.env.dev`, following the example in the default values file `.env.ce`.

A few noteworthy config options:

- `DEV_MODE`: Configure the api-server and celery-worker replicas to restart automatically upon changes to Python source files for rapid code iteration.
- `FORCE_INITIALIZATION`:  When running multiple replicas of the api-server, there is a semaphore file `.initializing_db` that is created to prevent a race condition during database provisioning. In development, you should only run one replica. The `FORCE_INITIALIZATION` env var prevents an errant initialization from leaving a stale semaphore file that blocks subsequent initializations.

### Module registration and manifests

The CE configuration is specified by `config/config.yaml`. Modules are "registered" for integration into the CE by adding an entry in the `config/config.yaml#/modules` section of the YAML document.

### Private Docker image registry credentials

If there are registered modules with images in private registries, the Docker credentials must be supplied using a standard JSON-formatted string in the `DOCKER_CONFIG_JSON` env var in a `.env.images` file. This will populate `$HOME/docker/config.json` in the Celery worker containers for pulling images from private image registries. Example:

```
DOCKER_CONFIG_JSON = '{ "auths": { "registry.gitlab.com": { "auth": "dXNlcm5hbWU6YWNjZXNzX3Rva2Vu" }}}'
```

where the `auth` value is the output of bash command:

```bash
echo -n 'username:access_token' | base64
```

If the registry is `registry.gitlab.com`, for example, then you would [create a dedicated access token](https://gitlab.com/-/user_settings/personal_access_tokens) with scopes `read_api, read_repository, read_registry`.

## Start the application

Launch the application with

```bash
bash docker/launch.sh dev up
```

By default you should be able to access the service by opening http://127.0.0.1:4000/ in your browser. Watch logs for the api-server with

```bash
bash docker/launch.sh dev logs
```

You will need to manually run the `docker compose logs` command if you want to watch other services, such as the Celery workers. Use the command echoed by the `launch.sh dev logs` script as a template, appending `celery-worker` for example.

To stop and destroy all containers, use 

```bash
bash docker/launch.sh dev down
```

If you need to start with a clean slate (e.g. an empty database), delete the persistent Docker volumes using:

```bash
bash docker/launch.sh dev destroy-volumes
```

You may need to delete the `app/calculation_engine/migrations` folder when starting fresh since no database migrations are necessary in that case.

## Develop CE code

When running locally, you can use the default `admin` user for testing since the OIDC authentication will not work out-of-the-box. The Python example below shows how to obtain an object for interacting with the CE API as well as an example showing how to list the jobs owned by that user:

```python
from calculation_engine_api import CalculationEngineApi
if __name__ == "__main__":
    api = CalculationEngineApi()
    api.job_list()
```

Celery tasks can be monitored at http://localhost:4000/flower/tasks.
The object storage bucket with files generated by the job can be viewed at http://localhost:9001/browser/calculation-engine.

Users that authenticate via OIDC will not have passwords associated with their Django user accounts. In that case, the user must obtain an API token by visiting http://localhost:4000/, logging in, and then visiting http://localhost:4000/api/v0/token.

## Unit tests

The suite of tests can be executing using the command below. Optionally, use the `destroy-volumes` option to ensure a consistent, empty initial state.

```bash
bash docker/launch.sh test destroy-volumes
bash docker/launch.sh test up
```

This runs the bare minimum services (api-server and database) to execute the `run/run_tests.sh` script via the `run/run_api_server.sh` entrypoint script, where the dedicated environment file `docker/.env.test` sets the `TEST_MODE` env var. 

```bash
```

## Staging deployment via Docker Compose

We have a dev/staging deployment running in Docker Compose on a virtual machine hosted on Jetstream2 for rapid collaborative code iteration and testing. It also acts as a staging area for proposed updates before applying the the production instance running in Kubernetes.

The Compose command is different for this staging instance deployment:

```bash
bash docker/launch.sh prod up
```