Maintenance Guide for FAI Local (On-Premises) Server

The on-premises servers that host the Ayyeka solution require monitoring and ongoing maintenance to ensure reliability and availability of the services.

Failure to adequately maintain the server might result in downtime, lost data, and poor performance.

Ayyeka is not responsible for maintenance carried out on on-premises sites, nor for any resulting failures or loss of data.
 

Contents

Server Infrastructure

Maintenance Guidelines

Managing Disk Space

Troubleshooting

 


Server Architecture Overview

The server software is built of microservices, each of which is in charge of a set of features and capabilities (for example, user interface, data processing, command processing, and so forth). The microservices are deployed and executed using Docker containers, a lightweight virtualization technology that improves software stability, scalability, and security. 

The microservices interact with each other to perform tasks and handle requests from users and devices. In addition, they use the following common infrastructure:

  • MySQL Database for storing the data collected by the devices, as well as all the metadata required to process the data, manage users and devices, and in general, maintain the application's state. 
  • Redis™ for caching frequently accessed data in-memory and system-wide synchronization
  • Rabbit MQ™ for communication between microservices and also for receiving data from the devices 

 

High availability of services

The on-premises software package is designed to run on a single server. If high availability is a requirement, appropriate architecture should be planned, based on the customer's requirements. The Support team is available to discuss and assist with planning, in collaboration with the system integrator and the customer’s IT staff. 

 

Recommended solutions might include:

  • Redundant internet connections
  • Redundant servers
  • High availability setup for MySQL, RabbitMQ and Redis

 

Maintenance Guidelines

Ayyeka recommends the following monitoring, maintenance, and resilience activities:

 

Monitoring of server and services

  • Regular monitoring and cleaning of disk space to accommodate the data that is collected in the database.  Disk usage depends on the number of devices, the amount of data collected per device, and other factors.
    • There must always be sufficient free disk space available to allow for growth, backups, and software upgrades. The minimum amount depends on your specific setup, but at least 10GB should be available at all times. 
    • Disk space availability is especially critical during software upgrades.
    • Disk usage needs to be monitored periodically to gauge the rate in which free space is consumed.  
  • Regular preventive maintenance and health checks on CPU and memory.
  • Regular health checks on hypervisor when hosted as a virtual machine (VM).
  • The following critical services should be monitored and running on their standard ports on the base Ubuntu operating system:
    • MySQL
    • RabbitMQ
    • Redis
    • DOCKER™
  • Docker containers must be monitored to ensure they are running at all times. Containers that are being restarted frequently might indicate an ongoing or evolving issue with one of the microservices. 

 

Ongoing maintenance of the server

  • Regular operating system and security updates.
  • Regular updates of the Ayyeka software on the on-premise server. Note that major updates are available about every two months, and minor updates are available approximately bi-weekly.
Installing updates might overwrite any changes that you made to configuration files. Therefore, it is highly recommended that you back up your system before doing any updates.
  • Backup of the server that can be used to restore the server in the event of a disaster.  Backups should be performed regularly and stored offline in a safe location. It is recommended to test backups at least quarterly.

    Disk Space Monitoring and Planning

    It is recommended to monitor the following directories regularly.

     

    Directory Name

    Description

    /var/ayyeka/ak-dumps

    What these files are: Backup of raw files received over HTTP

    Growth: It can grow over time, but a default retention period of 14 days is enforced.

    What you need to do: Make sure to have enough disk space to hold files for this retention period. 

     

    /var/ayyeka/protodumps

    What these files are: Backup of raw files received over MQTT

    Growth: It can grow over time, but a default retention period of 14 days is enforced. 

    What you need to do: Make sure to have enough disk space to hold files for this retention period.  

     

    /var/ayyeka/backup 

    What these files are: Backup of the database and configuration files

    Growth: Might grow over time, but a retention period of 7 days is enforced. 

    What you need to do:

    • Make sure to have enough disk space to hold files for this retention period.
    • Back up this directory to an off-server storage. 
    /var/lib/mysql

    What these files are: DB storage

    Growth: Will grow as the number of devices in the system increases and as data is collected

    What you need to do: Make sure to have enough disk space and add more storage space as required. 

    /var/lib/rabbitmq

    What these files are: RabbitMQ storage

    Growth: Should not grow. 

    What you need to do: Make sure it does not steadily increase.

    /var/lib/docker

    What these files are: Docker container images and logs

    Growth: Should not grow regularly, might grow after software upgrades.

    What you need to do: Make sure it does not steadily increase. If it does, use the following commands:

    • Check log sizes command line: 
    docker inspect --format='' $(docker ps -a -q) | sudo xargs -n 1 du -h
    • To delete log files: 
    docker inspect --format='' $(docker ps -a -q) | sudo xargs -n 1 truncate -s 0
    • Removing unused images: 
    docker image prune -a
    /var/log/ayyeka

    What these files are: Application server log files

    Growth: Grows as a function of the server's activity. A retention period of 7 days is enforced. 

    What you need to do: Make sure it does not steadily increase. Fast growth of logs and error logs might indicate a problem. 

    /var/lib/redis

    What these files are: Redis data

    Growth: Should not grow.

    What you need to do: Make sure it does not steadily increase.

     

     

    Troubleshooting

    You can get information about the Docker containers by using the following CLI commands: 

     

    Information you need

    Command
    List all running containers
    dps

    Show live resource usage statistics of all containers

    dstats 

    List all images that are locally stored with the docker engine

    docker image ls
    Stop a docker container
    dstop 
    [container name as it appears in the
    dps command]
    Start a docker container
    dstart  

    [container name as it appears in the dps command]

    Restart a docker container
    drestart  

    [container name as it appears in the dps command]