Upgrade a deployment

A deployment is a Helm release of the meridian-runtime chart. Upgrading it moves the release to a newer version of that chart, in place: the database migrates, every component restarts on the new image, and the data, the key and the configuration stay.

What people notice:

• The CLI stays connected. A delegation from meridian connect is kept in the deployment's database, so it survives the dashboard restarting, and nobody reconnects a terminal.
• Browsers sign in again. A browser's session is held in the dashboard's memory, so the dashboard's restart ends it.
• The database stays up. A database the deployment brought, and its plugin registry, run images of their own, and an upgrade that only moves the version does not restart them. The first upgrade from a chart that still labelled their pods with its version restarts them once more. The migration waits for the database to answer, up to five minutes, rather than failing at once.
• Launched plugins keep their sidecar. A plugin keeps the sidecar image it was launched with until it is launched again. See after an upgrade.

You apply an upgrade with your own cluster rights. Nothing in a deployment holds a right to change the cluster, and the dashboard does not upgrade itself.

To	Run or do
Upgrade to the latest published chart	meridian upgrade-deployment
Upgrade to a particular version	meridian upgrade-deployment --chart-version <v>
Change a chart value	meridian upgrade-deployment --set <key>=<value>, or -f <file>
Move to the HA profile, or back	meridian upgrade-deployment --set resilience.profile=ha (or =tolerant)
Turn on TLS inside the cluster	meridian upgrade-deployment --set tls.internal.enabled=true
Turn on page checksums in the database it brought	meridian upgrade-deployment --enable-checksums
Return to an earlier revision	meridian rollback-deployment
Upgrade from your own pipeline	Plain Helm, Argo CD or Flux, below
Upgrade the command line itself	meridian upgrade, which is a different thing

Take a backup before you upgrade, and copy it off the cluster: meridian backup --to <dir>, with backups set up. See Back up and restore a deployment. What an upgrade changes on the way, release by release, is in the release notes.

Upgrading to chart 0.1.538

Chart releases are numbered by the CI run that tested them, so the numbers jump; 0.1.538 is the current one. Upgrading to it keeps the counts a deployment runs: resilience.profile stays as it was, and TLS inside the cluster and the WAL archive stay off until you turn them on (below, and Back up and restore). Each store's writer reads its own records again once a week, at low priority (<store>.scrub.intervalDays, 7; 0 for never), and meridian status shows when it last did and anything it found. A database the deployment brought keeps what it was made with: meridian status says whether its page checksums are on, and --enable-checksums turns them on. With Helm 4, the upgrade passes --force-conflicts (see Plain Helm).

Each component connects as its own database login (meridian_lake, meridian_bor, meridian_instrument, meridian_street, meridian_conductor and meridian_dashboard), each query held to 5 seconds, so one component's slow query is refused on its own and never holds the database for the rest. The upgrade makes them in a database the deployment brought. For a database you run, its administrator makes them (see Install a deployment); until then the upgrade's check warns, naming each login missing with the statement that makes it, and each such component connects as the serving role, as before.

Load is refused, never crashed on. Each component works within a budget of its memory limit (memory.budgetPercent, 80) and, with it full, refuses a request for load with a time to ask again; past memory.shedPercent (90) it cancels its largest request. Each plugin has its share at its sidecar (sidecar.limits.inFlight, requestsPerSecond and answerMiB: 16, 400 and 32 MiB), past which only that plugin is refused; a launch can give one its own (see Its limits, shards and standby pod).

It also brings the Resilience page and meridian status --profile, a restore that chooses its timeline and --to-archive-end, and gaps and switches to a backup source on Data sources. A plugin on SDK 0.36.0 needs it.

Upgrading from a chart before 0.1.408

The upgrade to chart 0.1.408 adds each store's change log, the table a standby follows, and the stores' hot layers, each bounded within the component's memory limit (see Service levels). Every standby value defaults to 0, so a deployment keeps one member of each component until you ask for more. It also brings bonds and fixed income: the debt templates, pools' factors in the lake, and the book's fixed-income records. Plugins keep running as they are: one built on an older SDK registers and sends what it did; move it with meridian plugin migrate.

Upgrading from a chart before 0.1.313

The upgrade to chart 0.1.313 migrates the instrument store (a listed series' terms, and the company records), the conductor (the deployment's settings kept one record each, and starter packs) and the dashboard's notices, each adding to what is held. The book's snapshots change format, so each account's projection is rebuilt from its journal once; the journal is not rewritten. The settings home page gains Series request rule (series for an order asked for, standing wants not, 500 a day) and Stale reference actions permitted (refuse only); the Data sources page its Age tab; Browse its Starter packs view. Plugins keep running as they are: one built on SDK 0.32 registers and sends what it did, and reports no latency until it moves to SDK 0.33.0 (see Moving a plugin to a new release). An ems moved to SDK 0.33.0 reads Stale reference actions permitted, so a desk whose admin chose Warn refuses until a deployment admin permits Refuse or warn. The options measures price at the US Treasury's par yield curve, so a deployment running them launches a rates plugin recording it (see Rates).

Upgrading from a chart before chart 0.1.309 also rebuilds each account's projection from its journal once, as the book's snapshots change format, so every lot relieved before it gains its proceeds; the journal is not rewritten. The dashboard's settings move from a row of tabs to a grouped sidebar, every address unchanged, with FX pivot currencies under Deployment, reading USD, then EUR, until a deployment admin sets them. Placements made before it have no type and read as DMA. Plugins keep running as they are: one built on an older SDK registers and sends what it did, and one moved to SDK 0.32.0 or later records and reads rates, liveness and execution costs, and places by type (see Moving a plugin to a new release). From chart 0.1.311 a plugin whose catalogue states the licence's attribution terms registers, and rates resolve by their publisher's own codes (see The licence list). Upgrading from a chart before chart 0.1.308 also adds the analytics component: two Deployments, a coordinator and a worker, on an image of its own, ghcr.io/open-meridian/meridian-analytics, sized by the chart's analytics.coordinator and analytics.worker values, stateless, with no database and no volume, every pricing model built-in until a deployment admin routes one; and the book records the agents' guard on, once, as its own entry, a connectivity plugin reporting no environment for its session being live for agents. From one before chart 0.1.306, it starts the book's snapshots, one every bor.snapshotInterval entries (default 1,000), and gives the Plugins page Browse and Registries (see Registries); from one before chart 0.1.305, it runs that release's instrument migration (see Review provisional records and the migration). meridian status shows the release, each component and plugin, and the book's health after the upgrade.

To upgrade with the command line

meridian upgrade-deployment

It checks first, and changes nothing if a check fails:

• the cluster is reachable, and you may patch Deployments and create and delete Jobs in the namespace;
• your helm is 3.14 or newer, which an upgrade needs (see why);
• no node is under disk pressure. An upgrade pulls new images onto the node, which would take more of the disk it is short of, and its new pods would be scheduled nowhere. Free disk and wait for the pressure to lift; on a laptop VM, docker builder prune -a often does it. Low free disk is only reported, as worth, and does not stop it;
• the release exists and Helm holds it as deployed, not failed or pending-…;
• the version you are moving to is published, and is not older than the one installed.

If the release is at that version already, it says so and exits 0.

Then it shows what it will do, and asks:

Upgrading meridian in meridian:
  from  meridian-runtime 0.1.180 (revision 7), running ghcr.io/open-meridian/meridian-runtime:845bd06
  to    meridian-runtime 0.1.182, running ghcr.io/open-meridian/meridian-runtime:9c5d480

  helm upgrade meridian oci://ghcr.io/open-meridian/charts/meridian-runtime --version 0.1.182 --namespace meridian --reset-then-reuse-values --timeout 10m

Upgrade it? [y/N]

Where the deployment was installed with an archive for its edge plugins' older records, the plan says it is kept, as every value the deployment was given is:

  Its archive, /mnt/nas/meridian-archive on the cluster's node, is kept: the deployment's own values carry it.

If a node restart has left pods behind, the plan names them too, and the same answer covers removing them (see pods left over from a restart):

  Then it will remove 7 pods left over from a restart, which serve nothing:
    pod/meridian-meridian-runtime-broker-6d9c7b5f4-x2x8q
    ...

Answer y. It runs that helm upgrade, then waits, up to --timeout, for:

1. the new revision's migration Job to complete;
2. every Deployment and StatefulSet of the release to roll out;
3. every pod of theirs to run the image its template now names.

While it waits it shows where it is (see while it runs). When they have, it deletes the finished Jobs the release ran at earlier revisions and the pods left over from a restart, and reports:

Upgraded meridian in meridian: meridian-runtime 0.1.180 -> meridian-runtime 0.1.182, now revision 8.
Migrated: job/meridian-meridian-runtime-migrate-8 completed.

Every component on its new template, and ready:
  conductor  1/1  ghcr.io/open-meridian/meridian-runtime:9c5d480
  dashboard  1/1  ghcr.io/open-meridian/meridian-runtime:9c5d480
  ...

Cleaned up 2 finished Job(s) of earlier revisions: job/meridian-meridian-runtime-migrate-7, ...
Removed 7 pods left over from a restart: pod/meridian-meridian-runtime-broker-6d9c7b5f4-x2x8q, ...
Old ReplicaSets are left to the chart's revisionHistoryLimit.

A container that restarted while the upgrade ran is listed with the reason Kubernetes gives and the meridian logs <component> --previous that shows why. That is worth reading, and it is not a failure: a component that starts before its migration has finished exits and is restarted. meridian status shows each component's restarts and the last one's reason at any time, and meridian logs <component> what its running containers have printed.

While it runs

On a terminal, a block below what it has said is redrawn in place every half second: each step with its time, the total time, how many components are ready, and what it waits on now, and why.

Upgrading meridian in meridian  1m 04s
  ✓ checks                     3s
  ✓ plan and confirmation     12s
  ✓ apply                      8s
  ⠹ migration                 41s
  ⠹ components                41s  ready 4 of 7  ███████████░░░░░░░░░
  · cleanup
  waiting on conductor: 0 of 1 ready, waiting for its migration

A step is pending (·), in progress (a spinner), done (✓) or failed (✗). The migration and the components are waited for together: a component that starts before its migration has finished exits and is restarted, so while the migration runs, that is the reason given. Once the upgrade ends, the block stays with every step's time, above the report.

Piped to a file or run in CI, it writes a line as each step starts and ends, and every 20 seconds a line saying what it still waits for, so a log never looks stalled:

[3s] apply: started
[11s] apply: done in 8s
[11s] migration: started
[11s] components: started
[31s] still waiting, 4 of 7 components ready: conductor: 0 of 1 ready, waiting for its migration
[45s] migration: done in 34s
[1m 02s] components: done in 51s

Set NO_COLOR to turn the colour off.

Pods left over from a restart

When a node restarts, as a laptop's Kubernetes does when the laptop does, each pod it ran can be left behind stopped, as Failed or Succeeded, on the image it had then. Its Deployment has already started a new pod in its place, and nothing restarts or removes the old one. It serves nothing.

The command does not wait for such a pod: one stopped for good and made by a ReplicaSet its Deployment has since replaced. Waiting for it to run the new image would last until the timeout, since it never changes. It says so as it waits, and removes it in the cleanup, by name, with the release's other leftovers. A plugin's pods are treated the same way: a launched plugin carries the release's labels. A pod still pending, running or terminating on the old image is waited for.

--yes answers the question for a script that has already read the plan. --release, -n, --chart and --timeout are as for meridian up; see the command line reference.

After an upgrade: relaunch plugins

A launched plugin keeps the sidecar it was launched with until it is launched again, so after an upgrade it may still run beside the old version's sidecar. The report ends by naming each such plugin, and each plugin whose pod started while the upgrade was under way, when the old launcher may have launched it. For each it prints the commands that move it:

meridian plugin stop snaptrade
meridian plugin launch snaptrade <version> --instance snaptrade

For a plugin running live it prints meridian plugin dev --instance <instance> instead. The version is left as <version>: meridian plugin list gives it. It relaunches nothing itself. meridian doctor warns about the same plugins, as worth, at any time.

If it stops

Where it stops, it names what to run next, in meridian commands alone.

It says	What to do
a check stops	Nothing was changed. Each one says what to do.
the release is failed	meridian status says why, in Helm's words, on its Release line. Fix the cause, then meridian rollback-deployment returns it to its last deployed revision, and you can upgrade again.
the release is pending-upgrade or pending-install	Another Helm operation is running, or one was interrupted. If none is running, meridian rollback-deployment clears it.
the migration failed	It names the Job. Read its log with meridian logs migrate. The new revision stays applied. Read the log before rolling back: a migration that got part of the way may have changed what the older version reads.
still waiting after the timeout	It lists what it was waiting for, each with why. Nothing is rolled back. meridian status shows where each component is. Run it again with a longer --timeout: it is at the new version already, so it only waits.

To change a chart value

meridian upgrade-deployment --set backup.walArchive.enabled=true

--set <key>=<value> changes one of the chart's values in the same approved upgrade, read as Helm reads it: a.b=v, several split by commas, a.list[0]=v, a list as {x,y}, and true, false, null and whole numbers typed. Repeat it for more; each applies over every -f file. A value is an upgrade even at the chart version the deployment runs already: the version stays, and the value changes. At that version, where the values come out as they were, it says there is nothing to do.

The plan names each path that changes, never its value. The --set is shown in the helm upgrade line, except a value whose key names a secret, password, token, key, code or certificate, shown as <withheld>:

Upgrading meridian in meridian:
  from  meridian-runtime 0.1.538 (revision 3), running ghcr.io/open-meridian/meridian-runtime:aae501d
  to    meridian-runtime 0.1.538, running ghcr.io/open-meridian/meridian-runtime:aae501d

  Its values, with --set backup.walArchive.enabled merged over them (values not shown):
    backup.walArchive: set

  helm upgrade meridian oci://ghcr.io/open-meridian/charts/meridian-runtime --version 0.1.538 --namespace meridian --reset-then-reuse-values --timeout 10m --force-conflicts --set backup.walArchive.enabled=true

Upgrade it? [y/N]

Then it applies and waits as any upgrade does. --set goes with --chart-version too, to move the version and change a value under one approval.

To move to the HA profile, and back

resilience.profile turns every count of the HA configuration on at once: a standby for each writer, the database the deployment brought with two standbys under Patroni, the broker at three, and the dashboard and the analytics component at two (see Service levels). It needs the deployment's key in a Secret named by key.existingSecret; the chart refuses ha without one.

meridian upgrade-deployment --set resilience.profile=ha

A brought database moves under Patroni in the chart's two upgrades. The plan says so before it asks, and the one approval covers both:

  Its values, with --set resilience.profile merged over them (values not shown):
    resilience: set
  Its database moves under Patroni, which takes two upgrades, and this one approval covers both. The first moves its member 0 under Patroni alone, keeping its data; once that has rolled out and member 0 is ready, the second, with the same chart and values, adds its 2 standbys, which clone from it. Until member 0 is ready the database answers nobody.

It runs the first, waits for it as for any upgrade, then runs the second:

The first of two upgrades: the database moves under Patroni, member 0 alone.
Applied (1 of 2) as revision 5. Waiting for the migration, then every component on its new template (up to 10m).
...
Member 0 is ready under Patroni (revision 5). The second upgrade, with the same chart and values, adds its 2 standbys, which clone from it.
Applied (2 of 2) as revision 6. Waiting for the migration, then every component on its new template (up to 10m).
...
Upgraded meridian in meridian: meridian-runtime 0.1.538 -> meridian-runtime 0.1.538, now revision 6.

If the first stops, the second is not run, and it says so. Once member 0 is ready, run the same command again: it finds the database under Patroni alone, says that this upgrade is the second, and adds the standbys. If the second stops, the first stays applied, with the database under Patroni, member 0 alone and serving; the same command adds the standbys.

meridian status then lists the database's members:

Database:
  meridian-meridian-runtime-database  3 member(s), under Patroni, WAL archive on
    meridian-meridian-runtime-database-0  primary  ready
    meridian-meridian-runtime-database-1  standby  ready  0 B behind the primary
    meridian-meridian-runtime-database-2  standby  ready  0 B behind the primary
  page checksums on, full_page_writes on

Back to tolerant is one upgrade:

meridian upgrade-deployment --set resilience.profile=tolerant

Each count returns to one. A database that has moved under Patroni stays under it, with no standbys.

To turn on TLS inside the cluster

meridian upgrade-deployment --set tls.internal.enabled=true

The chart makes a certificate authority of its own and each server's certificate from it, kept across upgrades, and gives every client the authority's certificate. The deployment's own connections inside the cluster then use TLS, and a plain one is refused: the stores', the dashboard's and the Jobs' to the database the chart brings, every component's and sidecar's to the broker, the brokers' to each other, the database's standbys to their primary, and the dashboard's requests to each plugin. A database somebody else runs is then reached over TLS too, its server verified against the public roots and this authority, or against database.caSecret; one that offers no TLS is refused.

It is one upgrade at the version the deployment runs. While it rolls out, each launched plugin's pod may restart once. meridian status then says internal TLS on. It is off by default: a desktop's one node has none of these connections on a network.

To turn on page checksums

meridian upgrade-deployment --enable-checksums

With data checksums on, Postgres notices a page corrupted on its disk when it reads it. A database the chart makes now has them from the start. One made by an earlier chart has them off: meridian status says page checksums off, and meridian doctor says so as worth and names this command. Turning them on stops the database while pg_checksums reads every page, so it is a step of its own: it takes no --chart-version, -f, --set or --https, and no chart moves with it. Where checksums are on already, it says so and changes nothing.

It checks first, and stops, changing nothing, unless the release is deployed, it brought its database, the chart holds the step (0.1.538 does), and backups are set up, since it takes one first (see Turn backups on). Then it says the downtime, estimated from the database's size, and asks:

Turning on page checksums in meridian-meridian-runtime-database (meridian in meridian):
  the database  8.7 GiB on member 0's volume, 1 member
  downtime      about 7 minutes, an estimate: pg_checksums reads and writes every page, taken at 50 MiB/s, and stopping and starting everything takes a few minutes.
                Nothing answers until it is done.

  1. A backup, now, as `meridian backup` takes one. If it fails, nothing is changed.
  2. The release's CronJobs suspended (the replica comparison, the backup, the scrub), every component stopped, then the database.
  3. pg_checksums --enable on member 0's volume, in a Job from meridian-meridian-runtime-database-offline.
  4. The database started, then every component, each to the count it ran at, and each CronJob as it was.
  5. data_checksums read again, to say it is on.

Turn page checksums on? [y/N]

The estimate is a cautious one: the database's size at 50 MiB a second, plus four minutes to stop and start everything, plus the same again for each standby. What stops is everything: every component, the launched plugins among them, and the database. Nothing answers until it is done. The release's scheduled jobs, the backup and, with standbys, the replica comparison, are suspended while it is stopped, and each is given back as it was. Under Patroni the primary is moved to member 0 first; afterwards the standbys' volumes are deleted, and Patroni makes each again from member 0. At the end it reads the setting again:

Page checksums are on in meridian's database, meridian-meridian-runtime-database.
Backed up meridian to meridian-20261109T151322Z.meridian-backup (<bytes> bytes), on the deployment's backup volume.
  ...
  database-offline: page checksums enabled: ...

Where it stops:

It says	What it means
the backup failed	Nothing was changed.
stopping everything did not finish	The database was not changed, and everything was started again.
the Job failed	pg_checksums marks the database last, so it is as it was, without checksums, and everything was started again. The backup taken first is named.
the Job is still running	Everything stays stopped while the Job has the database's volume. Once it has ended, run meridian upgrade-deployment --enable-checksums again: it takes it from there, and starts everything.

While it has the deployment stopped, meridian status says so, since when, and the command that resumes it (see meridian status). It does not start while a restore has the deployment stopped, or while a Job of an earlier one still runs.

To roll back

meridian rollback-deployment

It returns the release to the last revision deployed before the newest; --to-revision <n> names another, as meridian status numbers them. It checks first, and changes nothing if a check fails. Then it shows the revision it rolls back from and the one it returns to, and asks:

This rolls meridian in meridian back:
  from revision 8, failed, chart meridian-runtime-0.1.… (app …)
  to   revision 7, chart meridian-runtime-0.1.… (app …), as a new revision
  Helm says of 8: …
The database is not rolled back with it. …
Roll it back? [y/N]

The database is not rolled back. What the newer revision's migration changed stays, and the older runtime may refuse a schema it does not know. Read meridian logs migrate first. If the data must go back too, restore the backup you took before the upgrade (see Restore a backup).

Answer y. It rolls back, then waits, up to --timeout, for that revision's migration if it runs one and for every component on its template, as an upgrade does. --yes answers for a script. See meridian rollback-deployment.

To upgrade from your own pipeline

Firms usually apply an upgrade through their own change control: a reviewed version bump, applied by a pipeline. The chart is the same one the command line applies, and it works that way too. Whatever applies it, do what the command line does:

Before: the release is deployed; the new version is published and is newer than the one installed.

The upgrade: the new chart's defaults, with the deployment's own values over them.

After:

1. Wait for the migration Job of the new revision, <release>-meridian-runtime-migrate-<revision>, to complete. If it fails, stop and read its log.
2. Wait for every Deployment and StatefulSet with the label app.kubernetes.io/instance=<release> to roll out.
3. Delete the finished Jobs of earlier revisions. Their names end with the revision that ran them. The Job ending in -key is kept: it runs once, at install.

kubectl rollout status does not wait for pods left over from a restart, so a pipeline is not held up by them. Deleting them is tidying, and optional.

Plain Helm

helm upgrade meridian oci://ghcr.io/open-meridian/charts/meridian-runtime \
  --version 0.1.182 --namespace meridian --reset-then-reuse-values

kubectl wait -n meridian --for=condition=complete --timeout=10m \
  job/meridian-meridian-runtime-migrate-8
kubectl rollout status -n meridian deployment -l app.kubernetes.io/instance=meridian --timeout=10m
kubectl rollout status -n meridian statefulset -l app.kubernetes.io/instance=meridian --timeout=10m

helm history meridian -n meridian gives the new revision, the number the migration Job ends with.

With Helm 4, add --force-conflicts, as meridian upgrade-deployment does: the chart's counts then replace the ones a first run or a restore set, as Helm 3 replaced them. A database moving under Patroni takes two upgrades with the same values, the second once kubectl rollout status statefulset/meridian-meridian-runtime-database says the first has rolled out; the chart's notes say so after the first.

Do not add --wait. Charts up to 0.1.182 mark an upgrade failed under --wait on one of their own setup resources, although the upgrade worked. Wait with kubectl as above.

Flux

A HelmRelease runs Helm itself. The values it holds are the deployment's own, and on each upgrade it applies them over the new chart's defaults, which is what --reset-then-reuse-values does. To upgrade, change the version in Git:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: meridian-runtime
  namespace: meridian
spec:
  interval: 1h
  url: oci://ghcr.io/open-meridian/charts/meridian-runtime
  ref:
    tag: 0.1.182          # the version; change this to upgrade
  layerSelector:
    mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
    operation: copy
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: meridian
  namespace: meridian
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: meridian-runtime
  upgrade:
    disableWait: true     # see Plain Helm: wait for the migration yourself
  values: {}              # the deployment's own values

Flux waits for the resources by default, which fails as --wait does on charts up to 0.1.182; disableWait turns that off. Check the migration Job and the rollouts afterwards, as above.

Argo CD

Argo CD renders the chart with helm template and applies what it renders. Its Application holds the deployment's own values, so there is nothing to reuse: each sync renders the new chart's defaults with those values. To upgrade, change targetRevision in Git to the new chart version.

Jobs named by revision

helm template renders every release as revision 1. The chart names its migration Job and its other per-revision Jobs by Helm's revision, so under Argo CD each upgrade renders a Job with the same name as the last one, and Kubernetes refuses to change a Job's template. Delete the finished migration Job before syncing an upgrade. Nothing else depends on it.

Why not --reuse-values

helm upgrade --reuse-values reuses the previous release's values, the old chart's defaults among them. The image tag is one of those defaults, so the upgrade applies the new chart's templates and keeps the old image: every pod restarts, and nothing changes version.

--reset-then-reuse-values starts from the new chart's defaults, the new image among them, and applies only the values the deployment was given over them. It needs Helm 3.14 or newer.

If the deployment's own values set image.tag, as meridian up --image does, that tag is one of its own values and an upgrade keeps it. meridian upgrade-deployment says so before it asks. Remove image.tag from the deployment's values if it should move with the chart.

Related

• Install a deployment
• Back up and restore a deployment
• Remove or start over
• Command line reference