GitHub/quartz
1
0
Fork 2
mirror of https://github.com/jackyzha0/quartz.git synced 2026-03-25 07:25:42 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity

Updated migration documentation

Browse Source
This commit is contained in:
acsok 2024-06-10 13:28:47 -04:00
parent e577aed431
commit 2b93309161
6 changed files with 279 additions and 167 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
content/VCS-Documentation/images/running services.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 133 KiB

97
content/VCS-Documentation/server-install/Database Migration Scripts.md Normal file
View File

@ -0,0 +1,97 @@
This directory contains scripts for migrating an SQLite database to a PostgreSQL database as well as setting up a new database with the server ID.
## Prerequisites
- Postgre installed and running
- Liquibase installed
- Execution of Liquibase scripts
- SQLite database file(s) to be migrated(for upgrades only)
## Setup
In `/usr/vcs/docker/migration-scripts` modify the `.pgpass` file in with the correct content: `hostname:port:database:username:password
For example: ```
```
localhost:5432:postgres:vcs:vcs
```
>Replace with the actual information of your Postgres database. If your password contains special characters, make sure to escape them with a backslash \\
Make the scripts are executable and set the correct permissions for the `.pgpass` file:
```
chmod +x database_migration.sh
chmod +x disable_constraints.sh
chmod +x create_db_dump.sh
chmod +x transform_db_dump.sh
chmod +x consolidate.sh
chmod +x update_server_id.sh
chmod 600 .pgpass
```
**Existing server upgrade only:**
For single server grab the `vcs.db` file, and rename it `serverID.db`(use the actual server id of the server, in case of `vcs.db-wall` and `vcs.db-shm` rename those with the server id as well)
For multi server gather all the `vcs.db` files and rename as mentioned in the single server step
Move the database file(s) to the scripts directory(`/usr/vcs/docker/migration-scripts`)
## Running the Migration
ServerID - server id for the server
PSQL_PATH - path to Postgres psql
Example: `docker exec -i artsentry-services-postgres-1 psql`
`artsentry-services-postgres-1` - name of the Docker container which can be found by running `docker ps`
#### New Server Install
Run the `update_server_id.sh` script with the `ServerID` and the `PSQL_PATH` parameter:
Example of running the script with Docker instance of Postgres:
```bash
./database_migration.sh 'server_id' 'docker exec -i artsentry-services-postgres-1 psql'
```
Example of running the script with local instance of Postgres:
```bash
./database_migration.sh 'server_id' '/opt/homebrew/Cellar/postgresql@16/16.2_1/bin/psql'
```
If you want to redirect the output to a different file, use the following command:
```bash
./database_migration.sh 'server_id' 'docker exec -i artsentry-services-postgres-1 psql' > migration.log
```
#### Existing Server Upgrade
To run the migration process, execute the `database_migration.sh` script with `PSQL_PATH` parameter:
Example of running the script with Docker instance of PostgreSQL:
```bash
./database_migration.sh 'docker exec -i artsentry-services-postgres-1 psql'
```
Example of running the script with local instance of PostgreSQL:
```bash
./database_migration.sh '/opt/homebrew/Cellar/postgresql@16/16.2_1/bin/psql'
```
If you want to redirect the output to a different file, use the following command:
```bash
./database_migration.sh 'docker exec -i artsentry-services-postgres-1 psql' > migration.log
```
## Scripts Description(FYI)
- **database_migration.sh**: Main script that orchestrates the migration process
- **disable_constraints.sh**: Disables constraints in the PostgreSQL database to prevent conflicts during data import
- **create_db_dump.sh**: Creates a dump of the SQLite database that can be imported into the PostgreSQL database
- **transform_db_dump.sh**: Transforms the SQLite dump to be compatible with Postgres
- **consolidate.sh**: Consolidates the transformed dump into the PostgreSQL database
- **update_server_id.sh**: Updates the server ID in the Postgres database after the migration or at new setup

161
content/VCS-Documentation/server-install/New install (single server).md
View File

@ -1,161 +0,0 @@
server 192.168.2.242
vcs:Asdf370)
*Required*
- vcs already installed (can be existing or clean install) for an existing installation the migration scripts will need to be ran to insert the data in sqlite into the postgres database.
To start the process of configuring postgres and setting up docker compose stop the vcs service using `systemctl stop vcs`
Server ssl needs to be disabled. Nginx will handle ssl.
disable ssl in `/cfg/settings.xml` set `WebServerSSL=false`
```xml
<General 
.
.
.
WebServerPort="80"
WebServerSSLPort="443"
WebServerSSL="false"
```
## Configure env.sh to use postgres
add the `-Ddatabase.type=postgres` to the `OTHER_JVM_OPTIONS` variable to use postgres as a database.
> `OTHER_JVM_OPTIONS` is not an environment variable. It is only read by `startvcs.sh`
also configure the following connection parameters.
>The `export` denotes that it is an environment variable
```shell
OTHER_JVM_OPTS="-Ddatabase.type=postgres <other-vm-args>"
#  datasource config postgres
export DATABASE_USER=vcs
export DATABASE_PASSWORD=vcs
export DATABASE_URL=jdbc:postgresql://localhost:5445/postgres
```
> NOTE: The `DATABASE_URL` should point to the postgres server. If Configuring multiple servers there will be a single database and the `localhost` should be replaced with the primary postgres servers ip when the postgres instance is not on the same machine as vcs.
When not root add the user to the docker group `sudo usermod -aG docker $USER`
# Running docker compose
Configure the `.env` `.env.local` and `.proxyGatewayConfig.json`. Adjust the server ip in each to point to the correct server.
## Configuration for compose
### Initialize docker and docker compsoe
The upgrade pack will contain a `docker.zip` and a `initialize_docker.sh` copy these two files to `/usr/vcs`
```bash
cd /usr/vcs
./initialize_docker.sh
```
The initialize script will install docker and the docker compose plugin. After Running the script there should be a new folder `/usr/vcs/compose-cfg` that contains the configuration for the docker compose environment. Configure the files as specified bellow
Configure the `.env` , `.env.local` and `proxyGatewayConfig.json`
The initialize_docker.sh script will copy the configuration files to /usr/vcs/compose-cfg edit the configuration files as specified in the next section.
### .env configuration
Change `VCS_SERVER_HTTP_URL=http://` to the vcs servers ip-address this should be the ip of the machine and also specify the port that vcs is listening on *(this should be an http url*).
> NOTE: VCS should be running on a higher port than 80 If vcs is not on 80 make sure to reflect that in this url. When running vcs on a higher port the existing swing client will talk directly to vcs so make sure to open that port in the firewall.
Set the `POSTGREST_VOLUME_PGDATA` and `POSTGRES_VOLUME_ARCHIVE` to a location where the database should store files. In this example it is in `/mnt/video00` this might need to change on production servers. The video drive or the root partition could be used.
```shell
APP_USER="vcs"
APP_PASSWORD="vcs"
DB_HOST="postgres"
DB_PORT="5432"
DB_NAME="postgres"
SSL_CERTS="./nginx_ssl_certs"
#Location of docker container configuration for multiserver-proxy and frontend
CONFIG_DIR="/usr/vcs/compose-cfg"
POSTGRES_VOLUME_PGDATA="/mnt/video00/dockerdata/postgres/pgdata"
POSTGRES_VOLUME_CONFIG="./postgres/config"
POSTGRES_VOLUME_ARCHIVE="/mnt/video00/dockerdata/postgres/archive"
VCS_SERVER_HTTP_URL="http://192.168.3.227:80" # cannot be localhost
```
### new-ui configuration (.env.local)
The new ui configuration is located in `.env.local`
Change the `NEXT_PUBLIC_API_SERVER` to the ip address of the vcs server.
`NEXT_PUBLIC_API_HOST` should be set to `http` if not going to use https for the new ui otherwise set this to `https`. `NEXT_PUBLIC_WS_SCHEMA` should be set to `wss` if using https or `ws` if using http. `NEXT_PUBLIC_API_PORT` should be set to the http or https port that nginx is listening on *(not the vcs http/https port)*.
`NEXT_PUBLIC_API_SUB_PATH` should be left blank if configuring for single server otherwise it will need to be `/gateway` to have it route requests to the mutiserver gateway. For more information on the multi server setup see //TODO:
Replace `<server-ip>` inside `NEXTAUTH_URL` with the ip address of the server. Also replace the `<nginxprt-1080>` with the same port used for the `NEXT_PUBLIC_API_PORT`
```shell
NEXT_PUBLIC_API_SERVER = <server-ip> #public ip used outside of the container to access the proxyGateway or vcs server (cannot be localhost as it will attempt to authenticate inside the container at vcs /api/v1/authenticate)
NEXT_PUBLIC_API_HOST = http
NEXT_PUBLIC_API_PORT = 1080
NEXT_PUBLIC_WS_SCHEMA = ws
NEXT_PUBLIC_API_SUB_PATH = ""
NEXT_PUBLIC_PROJECT = "Acuity-VCT" # "Acuity-VCT" | "Art Sentry",
NEXTAUTH_SECRET = my_ultra_secure_nextauth_secret
NEXTAUTH_URL = http://<server-ip>:<nginxport-1080>/artsentry/api/auth
NEXTAUTH_URL_INTERNAL = http://localhost:3000
```
### Gateway server config
The gateway server configuration is located inside `proxyGatewayConfig.json` this file contains a list of servers and their server-ids.
This is an example of configuring the gateway for two servers.
The only thing that should change in this file is the `servers` array. For a single server setup just define the ip of the single server in this json array. When the new ui is configured to go to `/gateway` this configuration will need to be correct in order to use multiserver in the new ui.
```json
{
"servers": [
{
"ip": "192.168.3.227",
"id": "VCS-DEV-INT-1"
},
{
"ip": "192.168.2.31",
"id": "290WPD2"
}
],
"secretKey": "secretKey",
"listeningPort": 3090,
"allowedOrigins": [
"*"
]
}
```
### Start the docker compose
### Login to github docker repo
the server will need to authenticate with the github package repo. Generate a jwt token that has access to artsentry and has package read access.
````bash
docker login ghcr.io -u USERNAME
````
change directories to `/usr/vcs/docker` make sure the .env file is located in `/usr/vcs/compose-cfg`
The following command will startup all the services that vcs requires
```bash
docker compose --env-file ../compose-cfg/.env up -d
```
to stop all the docker images run
```
docker compose --env-file ../compose-cfg/.env stop
```
since the `.env` file is not located next to the compose file it needs to be specified to each docker compose command.
to remove all the docker images
```bash
docker compose --env-file ../compose-cfg/.env down
```
>NOTE: Make sure the vcs server is not running until all the services are running and the database is properly configured in the vcs `env.sh` file
After the docker compose services are running start the vcs server
```bash
systemctl start vcs
```

175
content/VCS-Documentation/server-install/Postgres Setup (single server).md Normal file
View File

@ -0,0 +1,175 @@
*Required*
- VCS Server already installed (can be existing or clean install)
To start the process of configuring Postgres and setting up docker compose stop the vcs service using `systemctl stop vcs`
Server ssl needs to be disabled. Nginx will handle ssl.
Disable ssl in `/cfg/settings.xml`, set `WebServerSSL=false`
```xml
<General 
.
.
.
WebServerPort="80"
WebServerSSLPort="443"
WebServerSSL="false"
```
## Configure env.sh to use Postgres
Add the `-Ddatabase.type=postgres` to the `OTHER_JVM_OPTIONS` variable to use Postgres as a database.
> `OTHER_JVM_OPTIONS` is not an environment variable. It is only read by `startvcs.sh`
Also configure the following connection parameters:
>The `export` denotes that it is an environment variable
```shell
OTHER_JVM_OPTS="-Ddatabase.type=postgres <other-vm-args>"
#  datasource config postgres
export DATABASE_USER=vcs
export DATABASE_PASSWORD=vcs
export DATABASE_URL=jdbc:postgresql://localhost:5445/postgres
```
> NOTE: The `DATABASE_URL` should point to the Postgres server. If Configuring multiple servers there will be a single database and the `localhost` should be replaced with the primary Postgres servers ip when the Postgres instance is not on the same machine as vcs.
When not root add the user to the docker group `sudo usermod -aG docker $USER`
# Running docker compose
### Initialize docker and docker compose
**New Server Install**
`docker.zip` and `initialize_docker.sh` will be extracted into `/usr/vcs`(or install dir) at installation of VCSServer
**Existing Server Upgrade**
Copy `docker.zip` and `initialize_docker.sh` to `/usr/vcs`(or install dir) from the the upgrade_package
The initialize script will install docker and the docker compose plugin. Navigate to `/usr/vcs` and run the `initialize_docker.sh` script
```bash
cd /usr/vcs
./initialize_docker.sh
```
After Running the script there should be a new folder `/usr/vcs/compose-cfg` that contains the configuration for the docker compose environment. Configure `.env` , `.env.local` and `proxyGatewayConfig.json` as specified bellow
### Configure the docker compose environment(.env)
In `/usr/vcs/compose-cfg/.env` change `VCS_SERVER_HTTP_URL=http://` to the VCS servers ip-address, this should be the ip of the machine and also the port that the VCS is listening on *(this should be an http url, see example below).
> NOTE: It is recommended to run VCS on a higher port than 80. If VCS is not on 80 make sure to reflect that in this url. Also, when running VCS on a higher port the existing Swing client will talk directly to VCS so make sure to open that port in the firewall.
Set the `POSTGREST_VOLUME_PGDATA` and `POSTGRES_VOLUME_ARCHIVE` to a location where the database should store files. In this example it is in `/mnt/video00` this might need to change on production servers. The video drive or the root partition could be used.
```shell
APP_USER="vcs"
APP_PASSWORD="vcs"
DB_HOST="postgres"
DB_PORT="5432"
DB_NAME="postgres"
SSL_CERTS="./nginx_ssl_certs"
#Location of docker container configuration for multiserver-proxy and frontend
CONFIG_DIR="/usr/vcs/compose-cfg"
POSTGRES_VOLUME_PGDATA="/mnt/video00/dockerdata/postgres/pgdata"
POSTGRES_VOLUME_CONFIG="./postgres/config"
POSTGRES_VOLUME_ARCHIVE="/mnt/video00/dockerdata/postgres/archive"
VCS_SERVER_HTTP_URL="http://192.168.3.227:80" # cannot be localhost
```
### New-ui configuration (.env.local)
The new ui configuration is located in `.env.local`. In `/usr/vcs/compose-cfg/.env.local` change the `NEXT_PUBLIC_API_SERVER` to the ip address of the VCS server.
`NEXT_PUBLIC_API_HOST` should be set to `http` if not going to use https for the new ui otherwise set this to `https`. `NEXT_PUBLIC_WS_SCHEMA` should be set to `wss` if using https or `ws` if using http. `NEXT_PUBLIC_API_PORT` should be set to the http or https port that NGINX is listening on *(not the VCS http/https port)*. The ports are listed in the `/usr/vcs/docker/compose.yml` file.
Example:
```
  nginx-proxy:
    #    image: ghcr.io/acuity-vct/nginx-proxy:beta-1.2
    build:
      context: ./nginx
    ports:
      - "1080:80"
      - "10443:443"
```
`NEXT_PUBLIC_API_SUB_PATH` should be left blank if configuring for single server otherwise it will need to be `/gateway` to have it route requests to the mutiserver gateway. For more information on the multi server setup see //TODO:
Replace `<server-ip>` inside `NEXTAUTH_URL` with the ip address of the server. Also replace the `<nginxprt-1080>` with the same port used for the `NEXT_PUBLIC_API_PORT`
```shell
NEXT_PUBLIC_API_SERVER = <server-ip> #public ip used outside of the container to access the proxyGateway or vcs server (cannot be localhost as it will attempt to authenticate inside the container at vcs /api/v1/authenticate)
NEXT_PUBLIC_API_HOST = http
NEXT_PUBLIC_API_PORT = 1080
NEXT_PUBLIC_WS_SCHEMA = ws
NEXT_PUBLIC_API_SUB_PATH = ""
NEXT_PUBLIC_PROJECT = "Acuity-VCT" # "Acuity-VCT" | "Art Sentry",
NEXTAUTH_SECRET = my_ultra_secure_nextauth_secret
NEXTAUTH_URL = http://<server-ip>:<nginxport-1080>/artsentry/api/auth
NEXTAUTH_URL_INTERNAL = http://localhost:3000
```
### Gateway server config
The gateway server configuration is located inside `/usr/vcs/compose-cfg/proxyGatewayConfig.json` This file contains a list of servers and their server-ids.
This is an example of configuring the gateway for two servers. The only thing that should change in this file is the `servers` array. For a single server setup just define the ip of the single server in this json array. When the new ui is configured to go to `/gateway` this configuration will need to be correct in order to use multiserver in the new ui.
```json
{
"servers": [
{
"ip": "192.168.3.227",
"id": "VCS-DEV-INT-1"
},
{
"ip": "192.168.2.31",
"id": "290WPD2"
}
],
"secretKey": "secretKey",
"listeningPort": 3090,
"allowedOrigins": [
"*"
]
}
```
## Start the docker compose
### Login to GitHub docker repo
The server will need to authenticate with the GitHub package repo. Generate a jwt token that has access to Artsentry and has package read access.
````bash
docker login ghcr.io -u USERNAME
````
Change the directory to `/usr/vcs/docker`, make sure the `.env` file is located in `/usr/vcs/compose-cfg` and is configured as described above.
To start all the services that VCS requires, run:
```bash
docker compose --env-file ../compose-cfg/.env up -d
```
To stop all the docker images, run:
```
docker compose --env-file ../compose-cfg/.env stop
```
Note: Since the `.env` file is not located next to the compose file it needs to be specified to each docker compose command.
To remove all the docker images
```bash
docker compose --env-file ../compose-cfg/.env down
```
>NOTE: Make sure the VCS server is not running until all the services are running and the database is properly configured in the VCS `env.sh` file
To check all the running services see: [[running services.png]]
After the docker compose services are running start the VCS server
```bash
systemctl start vcs
```

11
content/VCS-Documentation/server-install/Server Install.md
View File

@ -1,9 +1,10 @@
# New server setup
1. install vcs
2. setup docker-compose/ configure postgres[[New install (single server)]]
3. database migration TODO: link to document
1. Install VCS Server as usual
2. Setup docker-compose and configure PostgreSQL: [[Postgres Setup (single server)]]
3. Execute migration-scripts: [[Database Migration Scripts]]
# Existing server setup
1. setup docker-compose / configure postgres [[New install (single server)]]
2. database migration for existing server
1. Upgrade VCS Server as usual (move content of /upgrade_package/brand to /usr/vcs)
2. Setup docker-compose and configure PostgreSQL [[Postgres Setup (single server)]]
4. Execute migration-scripts: [[Database Migration Scripts]]

2
content/index.md
View File

@ -10,4 +10,4 @@ git clone https://github.com/Acuity-vct/vcs-documentation
commit any documentation to `content/VCS-Documentation`
Documentation for [[New install (single server)]]
Documentation for [[Postgres Setup (single server)]]