/
State Software Maintenance

State Software Maintenance

Owned by Chad Carson
Last updated: Mar 07, 2025
6 min read

This catalog item performs a number of tasks to hosted State Software, and is the go-to item when something is not working correctly or needs updated.

This item performs four tasks to any applications you run it on:

  1. Restart the application

  2. Upgrade the application

  3. Apply any new configuration changes to the application

  4. Optimize the application’s Postgres Database

image-20250306-153021.png

A Quick Lesson in State Software Application Structure

To understand a little bit about what State Software Maintenance does, it helps to understand the structure of State Software applications in Kuberentes. Each hosted State Software application runs as two independent “pods” in Kubernetes.

  1. The Application (App) pod runs the user interface and the program logic.

  2. The Database (DB) pod stores the data, and runs the Postgres database and related tools.

If you have five applications deployed for a district, there will be 10 total pods.
This is how the applications appear to the MCOECN Datacenter team in Kubernetes

image-20250224-210256.png

There are two configuration files (YAML files) that store the configuration for each application. These are text-based markup files similar to XML files. Examples of items stored in the YAML are the application's name, the URL used to access the application, API keys and URLs to integrate applications, DUO configs, MiniOrange configs, and for some applications, Active Directory/LDAP configs.

The software images developed by SSDT for the applications and databases are stored in a central repository hosted by the MCOECN, named Artifactory. The URL/path to this central repository and application image is also stored in the YAML file. Artifactory URLs look similar to the following.

docker-images.ssdt-ohio.org/ess-app:latest

When an application is started up (both in Docker and Kubernetes), the application image is pulled down from the repository to the local machine via the defined URL, and started up as a container.

Pods themselves are non-permanent objects, they can be destroyed and recreated at will, and are quite often. Kubernetes has a structure called a Persistent Volume that is permanent disk space, where the postgres database is stored, and hence all of the USAS/USPS data. This is NOT destroyed, it remains persistent forever until explicitly deleted. Database pods attach to this persisent volume when they are created, which is how the data persists even when the database pod is deleted.

 

State Software Maintenance: Function One: Restart an Application

When you run State Software Maintenance on an application, it will completely restart the selected application, by performing the following steps:

  1. Shuts down/deletes the existing application pod

  2. Checks the database pod for any open connections or locks and clears them

  3. Shuts down/deletes the existing database pod

  4. Starts up a new database pod

  5. Starts up a new application pod

This restarts the application as completely as possible. When this is done, brand new pods will be running your application.

 

State Software Maintenance: Function Two: Upgrade an Application to the Latest Version

A side-effect of completely restarting an application, if a new version of the application has been released and placed in the central repository by SSDT since the last time the application was restarted, when the application restarts, it wlll pull the latest image version from the repository. Applications will not pull updates unless they are fully restarted. The steps from Function Two above can be revised as such:

  1. Shuts down/deletes the existing application pod

  2. Checks the database pod for any open connections or locks and clears them

  3. Shuts down/deletes the existing database pod

  4. Starts up a new database pod, which pulls the latest software version from the Artifactory repository

  5. Starts up a new application pod, which pulls the latest software version from the Artifactory repository

 

 

State Software Maintenance: Function Three: Apply New Configuration Changes

If any edits are made to the YAML file of an application (new API key, adding DUO configuration, etc), these do not take effect until they are applied. Some Aria tasks do the apply when they are run (like DUO), but some do not (deploying a new ESS will not restart USPS automatically, for example). Running State Software Maintenance applies the YAML configuration to the application when it restarts. The steps from Function Two above can be further revised as such:

  1. Shuts down/deletes the existing application pod

  2. Checks the database pod for any open connections or locks and clears them

  3. Shuts down/deletes the existing database pod

  4. Starts up a new database pod by applying YAML changes, which pulls the latest software version from the Artifactory repository

  5. Starts up a new application pod by applying YAML changes, which pulls the latest software version from the Artifactory repository

You would need to apply configuration changes after you have deployed a new application that requires API keys to integrate with other applications.

 

State Software Maintenance : Function Four : Postgres Database Optimization

The final function of State Software Maintenance is to perform Postgres Database Optimization.

Three functions should be done to Postgres databases on a regular basis:

  1. ANALYZE - Gathers stats on database tables to help create the most efficient data queries.

  2. REINDEX - Rebuilds table indexes to prevent corruption and bloat of empty data.

  3. VACUUM - Rebuilds tables to remove empty rows and unfragment the disk space.

These is done automatically once per month to every application as part of MCOECN State Software Hosting, so this should really not need to be done. However, the option exists to FORCE Postgres Database Optimization if you feel it may help. Note that if an application is DUE to have optimization run, when you run State Software Maintenance, it MAY run optimization anyway if it has been more than 30 days since the last run. If you choose to run Postgres Optimization, it would alter the steps once more:

 

  1. Shuts down/deletes the existing application pod

  2. Checks the database pod for any open connections or locks and clears them

  3. Shuts down/deletes the existing database pod

  4. Starts up a new database pod by applying YAML changes, which pulls the latest software version from the Artifactory repository

    1. Run Postgres Optimization (ANALYZE/REBUILD/VACUUM) here, before starting the application.

  5. Starts up a new application pod by applying YAML changes, which pulls the latest software version from the Artifactory repository

 

Running State Software Maintenance

  1. Click the State Software Maintenance card from the Service Broker Catalog

  2. Choose the type (Production/Test/Train) and the district you wish to apply maintenance to.

  3. Check the boxes beside the applications you want to run maintenance on..you can run this on multiple applications in the same request.

    image-20250307-141554.png

     

  4. You can optionally choose to run Postgres Optimization as described above.
    The last optimization dates for each application are shown in a table, so you know when it was last done. These optimizations are staggered so they are not all run at once, so dates vary from application to application. In the example below, since optimations have been run in the last month, they would not be run if we submitted this to run today (March 7, 2025). We can force it by checking the Force Postgres Database Optimization checkbox. You can schedule this task to run automatically at a time outside of business hours, or just set it run a couple of minutes in the future if you need it run right away.

    image-20250307-141643.png



 

 

Related content