Maintenance, backup and lockout

The services mostly keep themselves tidy: they delete old logs and rows, and the system restarts them after a failed start. A few things, however, are the operator's job: backing up the database, getting back in after a lockout, and replacing a secret now and then. This chapter goes through them.

Maintenance mode and the maintenance launchers

The account service's maintenance mode runs instead of serving: it performs a single action (user, password, role, key, peer token) and exits. It is the way back in when nobody can sign in to the web interface.

• The service does not need to be running, but the database does: maintenance uses the same database as the service and waits at most 10 seconds for it.
• It is safe next to a running service: the same shared write lock serialises it as it does two service instances, and maintenance never stops the running service.
• The usual route is the maintenance launchers: the AccountServer/maintenance folder has a Windows .cmd and a macOS and Linux .command launcher for every action. They run with a double-click, ask for whatever they need, and pause at the end so that a printed key or password stays readable.

The most common launchers:

Launcher	What it does
admin-password	a new password for the admin user, entered hidden, twice – the way out of a lockout
admin-enable	re-enables a disabled admin
user-list, role-list, key-list, peer-token-list	lists the users, the roles, the keys or the peer tokens
user-add, user-password	a new user, or a new password for any user
user-disable, user-enable, user-remove	disable, re-enable, delete
key-add, key-revoke	a new key for a user, or revoking a key by its identifier
peer-token-print, peer-token-reset, peer-token-remove	prints, replaces or removes a service's peer token
db-check	checks the database connection; writes nothing
db-backup, db-restore	backs up or restores a database on an external server

When nobody can sign in

If the administrator's password is lost, or the account has been disabled, run this on the account service's machine:

AccountServer\maintenance\admin-password

On macOS and Linux the same is AccountServer/maintenance/admin-password.command. It asks for the password twice, hidden; an empty Enter makes the program generate one and print it once. A password set this way must be changed at the first web sign-in. If the account is disabled, run admin-enable first.

Good to know:

• The password may only contain printable characters without accents, and must not start or end with a space – an accented letter could silently become a different password on the console. Dead keys on some keyboard layouts (~, ^) can also be mangled; if signing in still fails afterwards, ask for a generated password.
• If the administrator was not created as admin, user-list shows its name, and user-password sets a password for any account.
• On a machine with the service installed as a system service, the service reaches the database with its own, protected connection string, while the maintenance launcher uses your default database binding. If that points to a different (for example older) database, the action succeeds there. Run the db-check launcher first and see which database it prints.
• Setting a password invalidates every open browser session.
• The web interface does not let you disable or delete the last administrator able to sign in – so a lockout usually comes from a forgotten password.

Backing up the database

Users, roles, keys, the audit log and usage live in the account service's PostgreSQL database. There is no scheduled backup: backups are manual, and it is worth making one before every update.

# the product's own local database
AccountServer\managed-postgres.ps1 -Backup
# a database on an external server
AccountServer\maintenance\db-backup -Path D:\backup\account.backup

• The backup contains password hashes, so treat it as confidential; only its owner can read the file.
• The web sign-in key ring (the keys used to encrypt the sign-in cookies) is deliberately left out of the backup, so a stolen backup cannot be used to forge a sign-in.
• The services' own settings, the models and the logs are not in the database: the settings sit in the settings folder of the services' data root, and the machine's regular backup covers them.

Restoring

# the product's own local database
AccountServer\managed-postgres.ps1 -Restore D:\backup\account.backup
# a database on an external server
AccountServer\maintenance\db-restore -Name D:\backup\account.backup

1. Stop the account service (and all its instances). The restore warns about this.
2. The restore first loads the backup into a temporary database, checks it there, and only then swaps it in. Today's database is kept aside (with a timestamp added to its name) and can be deleted later with the printed command.
3. Start the account service again.
4. Every web sign-in is invalid: everyone has to sign in again.

The restored database gets a new generation (the database's unique identifier). If an account service instance was left running anyway, it stops on seeing this – simply start it again. Peer tokens live in the database, so after a restore the services carry on with the same token. A brand-new, empty database, however, does not know the old peer tokens: the services then reject every request after about a minute and a half, and AccountServer\setup-account-peers -Replace gives them new ones (restart the services afterwards).

Replacing a peer token

A service's peer token needs replacing if it has leaked, if it was lost (it cannot be shown, since only its hash is stored), or if the database is new.

1. On the account service's Keys page, press Replace in the service's row – or run the peer-token-reset launcher. The old peer token is invalid immediately.
2. On the service's machine, put the new secret into the named environment variable (for example LCP_ACCOUNT_TOKEN_CAPTIONSERVER); the dialog after issuing also prints the command to copy. If the account service runs on the same machine, AccountServer\setup-account-peers -Replace does it for all three services at once.
3. Restart the service. For a service installed as a system service, run the installer again (-InstallService), because it copies the secret into its own protected folder.

Until the new value is on the machine, the service gets no authorization snapshot, and with a removed peer token it rejects every key-borne request after three refusals (about a minute and a half).

When a saved setting stops it from starting

The caption service, the translator and the voice service take over a value saved on the web interface's Settings page at the next start, and it is stronger than the command line. If a saved value keeps the service from starting, start it once with the --ignore-saved switch (the caption service's launcher passes switches written at the end through unchanged: .\start-captionserver.ps1 --ignore-saved). The saved layer is skipped for that one start, but the saved values are not lost: the faulty value can be fixed or reset on the Settings page. This switch can only be given on the command line, not from a configuration file.

What cleans up on its own

What	How long it stays	When it is deleted
the services' log files	30 days	when the service starts
audit log rows	400 days	daily, as a background job of the account service
raw usage rows	62 days	daily; the daily summary stays
the daily usage summary	permanently	–
the Connections page's request history	the last 200 requests	when the service stops

Log cleanup is not continuous: it runs once when the service starts. A service running for months therefore keeps its old logs until the next restart. Files another program currently holds open are skipped.

Where next

• The pages of the account service
• Running as a service: Windows, Linux, macOS
• Troubleshooting