The main usage load for disks, on which the system is installed on, is the generation of graphs in 5 minute intervals. For each client at minimum 2 graphs are created. For the routers at minimum 5 graphs are created in relation to the count of interfaces. Graphs are stored in binary form on the HDD and take up to 500KB of space. Therefore in practice the system needs to work with a large number of small files in which every 5 minutes informations about transmitted data and measured values are written. For a larger number of graphs SAS disk drives are recommended. SAS drives are significant faster, in working with small files, then SATA drives. Basically it is about the IOPS that the disks are able to provide to the system. In practice SATA disks are able to provide up-to 100 IOPS , SAS on the other hand are able to provide up-to 150-180 IOPS (depending on whether it is a 10k or 15k disk). The sum of IOPS can only be increased by the number of disks in RAID or by installing SSD disks for system graphing. IOPS is the main limitation for working with graphs. If the sum of IOPS is insufficient, the system latency is going to be increased and in some cases sites may load slower because the system is waiting for the HDD to be able to read and write. In this case an increase of RAM or CPU upgrade will not help. The only possibility is to increase the number of disks ( or replace the slow SATA drives with faster 15k SAS drives ).

• It is now possible to move generated graphs to a RAMDISK. A Ramdisk is a self-contained part in memory, that behaves like a disk but Access times are several times faster than a conventional HDD, because it only uses RAM memory to store the data. If graph generation is run within RAMDISKS the IOPS and the disk usage will be significantly reduced. This is the most noticeable in large installations ( 1000 clients or more). In this cases we recommend moving graphs to a RAMDISK.

• To work with RAMDISK, run this utility

/usr/local/script/ispadmin/rrd_ramdisk.pl

• To display a help message, run it without any parameters

• Usage:

/usr/local/script/ispadmin/rrd_ramdisk.pl initialize - Initialize RAMDISK

/usr/local/script/ispadmin/rrd_ramdisk.pl destroy - Destroy RAMDISK permanently

/usr/local/script/ispadmin/rrd_ramdisk.pl start [force] - Start RAMDISK

/usr/local/script/ispadmin/rrd_ramdisk.pl stop - Stop RAMDISK

/usr/local/script/ispadmin/rrd_ramdisk.pl save [force] - Save RAMDISK data to backup

/usr/local/script/ispadmin/rrd_ramdisk.pl check - Check RAMDISK usage

/usr/local/script/ispadmin/rrd_ramdisk.pl set - Set RAMDISK in MByte

/usr/local/script/ispadmin/rrd_ramdisk.pl restore [backup.file] - Restore RAMDISK from last backup.file

• You may use the following parameters:
  • initialize - Initializes the RAMDISK and moves the data to the RAMDISK.
  • destroy - Unistalls the RAMDISK and moves the data back to the drive.
  • start - For an automatic start after booting. Use [force] to initialize RAMDISK even if the size of the restored backup is smaller by more than 5% than the previous ramdisk size, or if RAMDISK initialization was not successful after server reboot, and you need to initialize RAMDISK from the console.
  • stop - Makes an automatic backup of the data in RAMDISK, copies the backup to a Hardrive, and stops RAMDISK. !!! We strongly recommend that you perform it every time you restart/switch off the system !!!
  • save - Makes a backup of data of the RAMDISK to a Hardrive - performed automatically once a day, 10 backups are done retrospectively, and you may restore them later. Moreover, 2 monthly backups (as of 28th of each month) are stored.
    (It backups into /usr/local/script/ispadmin/rrd_data_backup/. Use [force] to make a backup of ramdisk which is smaller than the previous ramdisk in size, e.g. if you deliberately delete more files with graphs. A check on 5%-difference compared to the original size is performed.)
  • check - Checks the utilization of the RAMDISK and if needed, it automatically increases the RAMDISK.
  • set - Set RAMDISK size in MB to increase it according to your needs.
  • restore - Insert name of backup file from backup directory for rrd ramdisk to restore a daily backup of rrd data. If no name is inserted, the system will restore the last known backup from this directory. You may use data restore if you loose it, or the data is incomplete. The only must is initialized RAMDISK.

• When RAMDISK is installed, you need to run the following command:

/usr/local/script/ispadmin/rrd_ramdisk.pl initialize

• • This initializes and configures all that is needed. After rebooting the sytem it installs the RAMDISK again and the data are moved on to it.
  • Be aware that the data on a RAMDISK is saved in RAM and if a server malfunction occurs ( powersuply malfunction ) all data are lost. In the case of correct restart, the system automatically backs up data to the Hardrive an moves it onto the RAMDISK after booting. The system also makes an automatic backup onto a hardrive once a day. So in case of power supply outage, one-day data only is lost.
  • For RAMDISK creation it is absolutely necessary to have a sufficient amount of RAM. The maximum amount of RAM that can be used for the RAMDISK is 25%.
  • The necessary amount of RAM is calculated dynamically according to occupied disk space (mostly 510Mb - 5GB). If the calculation for RAM is resulting in a larger amount then 25%, the RAMDISK is not created and it is necessary to upgrade the amount of RAM. If RAMDISK capacity is more than 80% full, the system will try to increase the disk automatically (if RAM is sufficient). If it fails, you may increase RAMDISK manually using set option.

A) Temporary setting of IP address (until next Linux restart).
Just add the following two commands for settings of IP/mask and default gateway:

B) Permanent setting of IP address
Modify the current IP address to the one you want:

Caching-only is ready after installation of ISPadmin. It is sufficient to use IP of ISPadmin as DNS server, and the system will then contact DNS servers in the Internet.
Type 0.0.0.0/0; (or type specific networks for which the DNS server should serve as cache) in /etc/bind/named.conf.options. We recommend that you list specific networks for which you want to use the ISPadmin as DNS server. If you allow to use ISPadmin for all networks (0.0.0.0/0) its services may be abused by anybody for e.g. DDOS attacks.

nano /etc/bind/named.conf.options

allow-query { 192.168.0.0/16; 10.0.0.0/8; 127.0.0.1; 0.0.0.0/0; };

Then restart bind

/etc/init.d/bind9 restart

If ISPadmin is part of internal network, outcoming ports 53/TCP and 53/UDP must be enabled in a terminal router.

nano /etc/resolv.conf

nameserver 127.0.0.1

nameserver IP_DNS_of_used_server

Set DNS servers from the Internet here. If your customers will use ISPadmin as DNS server, the system will work as Cache, and DNS queries will be translated by servers listed here. If IP address of name server is unknown, use Google name server (8.8.8.8).

By default, the SMTP server is configured in the way that it is only possible to send mail from localhost, so the server may be put onto a public IP address and nobody will be able to send their mail through it (spam). So ISPadmin does not send mail out (only locally).
Mail server configuration is done in /etc/postfix/main.cf:

nano /etc/postfix/main.cf

Postfix service must be restarted every time you modify configuration file:

/etc/init.d/postfix restart

ISPadmin has an automatic data back-up functionality. It is possible to store the backups locally (SW/HW RAID), but also on a remote repository (FTP, NFS, CIFS). It is advisable to store local backups on a partition different from the system one. If the system disk was fully used by backups, it might lead to a damage of the system.

Backup settings

It is done in Settings System Settings Backups.

System and router backup encryption

Router backup encryption is off after installation so you can turn it on by running an external script from linux console under root rights.

You can change/set the password with the following script:

/usr/local/script/ispadmin/ispadmin_backup_crypt.pl

If you set password value to -1 encryption will be turned off and it won´t be performed at next backuping.
You will be asked to provide email address during password setup/change. A message about changed password for backups will be sent to this email with both, old and new one. Do not delete this email or at least print out its content and thoroughly keep it somewhere since the password will be needed for restoring the data from a backup. There is no other way of getting or restoring it! Check if emails are sent from the system prior to changing the password. If sending fails no message with new and old passwords will be delivered to you.

System and router backups

You can create a backup and search already existing ones in Other Backups.

Manual system backup

You can run manual ISPadmin backup with

ispadmin_backup backup all

This command will execute a complete system backup into /data/backup/incremental/. The whole backup usually takes between 10 to 30 minutes depending on the number of clients, or the amount of backup data. If you have e.g. 8GB of backups and big server load, backups may take longer. However, you have to allow for a slowdown of a web interface during backup.

The information contained in this manual pertains to the migration of ISPadmin only. It does not pertain to the migration of any other product.

Prepare new server

Download current ISO file and follow instructions on how to install ISPadmin.

Read more ...

1. Turn off services on old and new server

You have to turn off Apache, Cron and Nagios services on both servers so that no changes of data are made in both, system and database. You also prevent users from logging in.

2. Export database from old server

Create export directory

mkdir /data/backup/export

Get database password

cat /data/support/ispadmin/config/config.php | grep PASS | cut -f 3

ISPadmin database export - you will need password from previous command

mysqldump --add-drop-table --quote-names ispadmin -u ispadmin -p > /data/backup/export/ispadmin.sql

3. Copy data to new server

Rename the following directories on new server and create export directory. This way you will get a backup of original files from new installation.

mv /data/support/ispadmin/ /data/support/ispadmin_orig/
mv /usr/local/script/ispadmin/ /usr/local/script/ispadmin_orig/
mkdir /data/backup/export/

Allow old server IP

iptables -I INPUT -p tcp -s ip-stary-server --dport 22 -j ACCEPT

Recommendation: If you want to save time and transferred data we recommend you to delete graphs of client data from past years. They are in /data/support/ispadmin/client_images/, divided into years. Ideally, leave current year and delete the rest, if you don´t need it.

Copy data from old to new server Do this on the old server:

rsync -av /data/support/ispadmin/ root@ip-new-server:/data/support/ispadmin/
rsync -av /usr/local/script/ispadmin/ root@ip-new-server:/usr/local/script/ispadmin/
rsync -av /etc/ root@ip-new-server:/data/backup/export/etc/
scp /data/backup/export/ispadmin.sql root@ip-new-server:/data/backup/export/

4. Import database

  rsync daemon must be running on the new server! If not, start it with service rsync start, or install it with apt-get install rsync.

Import database dump to the new server. Default password is isp123.

mysql ispadmin -u ispadmin -p < /data/backup/export/ispadmin.sql

5. Check settings on new server

Database

Get database password transferred from the old server.

cat /data/support/ispadmin/config/config.php | grep PASS | cut -f 3

Change database password to correspond with recovered password.

/usr/local/script/ispadmin/ispadmin_change_pass.pl ispadmin ispadmin DB_password

Change DB root password to a new one, and write it down.

/usr/local/script/ispadmin/ispadmin_change_pass.pl mysql_root ispadmin new-password

6. Update to latest stable version

If your support is paid and you don´t have the latest stable version, run update.

ispadmin_update stable

If the update does not work and the following message appears:

HTTP request sent, awaiting response... 404 Not Found
2017-02-21 13:09:30 ERROR 404: Not Found.
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

use this command:

cp -av /usr/local/script/ispadmin_orig/ispadmin_update* /usr/local/script/ispadmin/

If the command does not help, please proceed according to the instructions that can be found here.

7. Turn of services

8. Finish individual configuration

You have all important configuration files in /data/backup/export/etc/ (httpd.conf, hostname, hosts, rc.local). You can copy them to /etc to restore configuration e.g. on your Apache web server from the old server. Which files to use or not depends on the situation - if you are directing the new server to the same IP addresses, or you want it to have the same hostname etc.

For reasons of compatibility and availability of future versions of ISPadmin, you have to maintain your Debian Linux version updated. This article will show you how.

First you need to find out which Linux version you are currently using. To do so, use this command.

cat /etc/debian_version

You may expect four possible scenarios based on the result of the command.

1) 7.x Current version of ISPadmin, no need for any updates System support from Debian side was announced until May, 2018.

2) 6.x.x If your version is 6, you may continue using the system, however, system updates for Debian 6 will end in February, 2016. It means no security patches will be issued, among others, and you will have to look after the system security yourself. If you want to switch to newer version, we recommend system migration instead of sole distribution upgrade.

3) 5.0.x (x represents number between 0 and 10) In this case just follow the instructions on update of debian 5 to debian 6 below.

4) 4.0 If your version is Debian 4.0, you will not be able to update Linux version without serious problems so you will need to re-install Linux from the installation DVD available at http://download.ispadmin.eu, and migrate data afterwards. You can have the migration done by our technical support, or do it yourself according to the migration instructions.

Update of Debian 5 to Debian 6

1) Enter sources to /etc/apt/sources.list as follows:

nano /etc/apt/sources.list

2) Preparation for Linux update

3) Upgrade from Linux 5 Lenny to Linux 6 Squeeze

DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" --force-yes -fuy dist-upgrade

4) Download of utilities necessary for update of Apache and Nagios and their application

sed 's/backup\//backup\/:\/usr\/share\/nagios3\/htdocs\//' /etc/apache2/httpd.conf >/tmp/httpd.conf
mv /tmp/httpd.conf /etc/apache2/httpd.conf

5) Restart Apach and Nagios

6) If update caused charset encoding problems in web interface, modify /data/support/ispadmin/config/config.php and /data/support/ispadmin_support/config/config.php - add row MYSQL_CHARSET=latin1

nano /data/support/ispadmin/config/config.php

MYSQL_CHARSET=latin1

ISPadmin does not currently support IPv6, it only supports IPv4. It is, of course, possible to implement IPv6 support should such need become globally important.

Run htop command from the Linux console to determine the problems. With this command, you can easily find out if your system is overloaded for reasons of insufficient CPU performance, small RAM, or whether processes wait too long for I/O operations with the system being paralysed by slow hard disk.

• Insufficient RAM amount

Problem	Too small RAM is installed, hence swapping to hard disk which decreases server performance.
Solution	Add RAM to server.

• Slow hard drive

Problem	In htop, you can see high values in load system section, yet graphs show sufficient free RAM and CPU cores are not loaded at 100% - most likely it is a problem with insufficient write/read speed from system hard disk.
Solution	Replace hard disk with SAS/SSD.

• Small CPU performance

Problem	The processor cannot deal with common tasks and keeps running at 100%.
Solution	Buy more powerful server.

If you have your ISPadmin located on private network and wish to access it from the Internet, the easiest option for you is to set 1:1 NAT on public IP address to private address of ISPadmin.
However, we do not recommend it since all ports and services on the server would be subject to DoS and BruteForce attacks etc.
Otherwise you have to redirect selected ports to private address of ISPadmin:

The system is typically connected to the network by one network card only. Both public, and private IP address may be used. Should you, however, wish to separate access from the Internet and from the private network, you can connect the system via 2 network cards. On the first one, there will be the public IP (the access will be allowed from selected number of IP addresses on the firewall due to security), and the second one will be connected into the private network (e.g. client portal).

Do the configuration from linux console by editing /etc/rc.local:

ifconfig eth1 192.168.168.100 netmask 255.255.255.0 route add default gw 192.168.168.1
ifconfig eth0 8.8.8.8 netmask 255.255.255.0 route add default gw 8.8.8.1

By default, the SMTP server is configured in the way that it is only possible to send mail from localhost, so the server may be put onto a public IP address and nobody will be able to send their mail through it (no open relay).

Switch to new system version using the following command from command line (login via SSH):

ispadmin_update stable - update to the latest stable version

ispadmin_update beta - update to the latest beta version

You will run update script which downloads and installs the latest update of ISPadmin automatically. System administrators should not be logged in during update, since database maintenance and restart of webserver are carried out. The update is done usually within minutes based on number of clients and server perfomance. You must wait until the end of the update or you may experience problems with the system.

Type ispadmin_update to display help.

ispadmin_update
ISP ADMIN UPDATER
(c) NET service solution, s.r.o.
Downloading version description....
Latest version:
ISP admin STABLE version: 4.17
ISP admin BETA version:
Which version are you going to install ?
Type for
STABLE VERSION ( 4.17 )
ispadmin_update stable
BETA VERSION ( - )
ispadmin_update beta
exiting.......
admin.ispadmin.cz:~#

If technical support cannot be verified, most likely it is due to DNS malfunction. In ISPadmin, DNS servers are located in /etc/resolv.conf. Check their accessibility and functioning, or enter different DNS servers, if needed.

nano /etc/resolv.conf

Malfunctioning web server could be another reason. Restarting it should resolve the issue:

 /etc/init.d/apache2 restart

Another error example is that ISPadmin server cannot contact our server with updates on port 80. In this case, verify communication at IP 95.173.193.60.

It may happen that during a "hard restart" of the server, some of the files containing the system database tables may be damaged (consider using UPS system). As a result of it, links between contacts and their services may occur, or similar problems.

If it happens you will receive error notifications on corrupted tables in the system.

Thus if you encounter the notifications while working in the system, login to the command line environment and enter the following command for repairing the table:

It is obvious from the following message that a database table has been corrupted: Table './ispadmin/visited_pages' is marked as crashed...
Use the corrupted table name visited_pages in the following command to fix it:

• First stop the database

/etc/init.d/mysql stop

• Reparation

myisamchk -o /var/lib/mysql/ispadmin/corrupted_table.MYI

Start the database again

/etc/init.d/mysql start

You will be able to rectify possible errors according to the example. However, it may happen that the corruption is non-repairable, and you will need to restore data from last backup. Contact technical support, or follow these instructions to restore system database.

There is a warning message function in the system that warns an administrator about a corrupted table right after login to allow for the soonest rectification possible.

You have to prevent these situations and rectify problems as soon as they arise.

You can try two ways:

First procedure

1. Insert installation DVD of ISPadmin and wait for command line to be displayed.

2. Mount system disk and switch to installation repair environment.

mount /dev/[hda1|sda1|md0] /hd
chroot /hd

3. And then change password as usual, i.e. enter it 2x.

passwd

4. Exit chroot environment and unmout disk

exit
umount /hd

5. Eject installation DVD and restart server

eject
reboot -f

Second procedure

Mount system partition and modify /etc/shadow and /etc/passwd so that root password is blank.

Enable booting from DVD drive on your computer and boot life Linux from DVD to graphical interface. Mount root of partition the password of which we are trying to recover. Search file:

/etc/passwd

Search row:

root:x:0:0:root:/root:/bin/bash

Delete letter x in it so that the row looks like this:

root::0:0:root:/root:/bin/bash

Leave remaining rows untouched and save file.

Another file to be modified is:

/etc/shadow

Search row starting with root

root:$2a$05$e7K5MayhjHODCvu6qFabq.H3QUucC0Osjas1SkbKnTlhacoW/mwFu:13528::::::

Remove all characters and leave this:

root:::: (word “root” and 4 colons)

Leave remaining rows untouched and save file.

Eject DVD and restart system. Then boot the system with forgotten password of root user. Log in as usual user and open terminal to enter

su

Now you are logged in as root and you can run this command

passwd

This command will define a new password for root user!

System variables are used for replacing items such as client's name, address, tariff etc. with real client values into emails, SMS, invoices, reminders, contracts etc. This way you will be able to use one template to create many personalized messages, emails, and more. The template may be used for a certain client group/all clients later.

System variables are divided into categories, e.g. Client, Internet, Invoices etc., and are being pendingly added, hence no overview can be listed here. You will be able to see complete list of the variables every time you see   List of system variables.

Most commonly used for: Settings Contracts My Templates, Settings Other Email templates / SMS templates, Invoicing Settings Templates Reminders

You may also add your own variables into the system.

Read more ...

Nano text editor is always used in this documentation. If you don’t like it you can use a simple editor mcedit which is part of a well-known file manager mc - Midnight Commander. 

mcedit /path/to/file

• F2 - save changes
• F10 - exit without changes Y/N

nano /path/to/file

• Ctrl + x then y - save changes and exit
• Ctrl + x then n - exit without changes

We recommend using htop program for easier monitoring of running process instead of common top.
With htop you can easily monitor use of system resources, load, uptime, but most importantly kill running processes in bulk.

If you need to find out if a certain process is running use the following command:

ps ax | grep process_name

E.g. you want to know if Nagios is running.

As you can see, Nagios is running under PID (Proces ID) 30500.

Restart of MySQL service

/etc/init.d/mysqld restart

Recovery of database password

cat /data/support/ispadmin/config/config.php | grep DBPASS

Database table correction

Some files with system database tables may be damaged after “hard restart” of server.
When you restart MySQL database an error message is displaying a damaged table. Thus if you encounter the notifications while working in the system, login to the command line environment and enter the following command for repairing the table:

First stop the database

/etc/init.d/mysql stop

Database table correction

myisamchk -o -f /var/lib/mysql/ispadmin/CRASHED_TABLE.MYI

/etc/init.d/mysql start

Or you can repair tables when MySQL is running with

mysqlcheck -u ispadmin -p ispadmin

You will be asked for database password, then a complete database check will be performed, including all tables and their reparation in case of minor errors.

Repair a certain database with

mysqlcheck -r -u ispadmin -p ispadmin table_name

"Europe/Prague” timezone is set by default. To change it, do the following:

Use command dpkg-reconfigure tzdata and select your option, e.g. Atlantic/Bermuda.

If time is correct timezone setup in Linux is done. Now you have to change timezone in PHP to have correct time in web interface, too.
Set it up in date.timezone item in /etc/php5/apache2/php.ini a /etc/php5/cli/php.ini. You will find the list of available timezones at: http://php.net/manual/en/timezones.php.

nano /etc/php5/apache2/php.ini

date.timezone = "America/Argentina/San_Juan"

Restart apache.

service apache2 restart

No, it cannot. ISPadmin is based on Debian Linux.

More on RAM disk here.

Starting RAM disk

If ramdisk is disconnected due to server outage or unexpected situation, you can start it by running the following script from ISPadmin console:

/usr/local/script/ispadmin/rrd_ramdisk.pl start force

This way you will start ramdisk and restore data from the last backup. Use [force] option to restore the data even if it is smaller than the last known data.

Restoring RRD data from backup

ISPadmin creates backups of graphs stored in ramdisk automatically once a day. 10 backups are created back.

If the data is corrupted and ISPadmin detects it (change in data size by more than 5% compared to the previous size, or disconnected disk), you can restore data to ramdisk, if initiated, or to the original directory with rrd data:

/usr/local/script/ispadmin/rrd_ramdisk.pl restore -- restores data from last backup

If the recovery of the ramdisk data does not fix your problem, try having a look at the list of backups and choosing the largest one. Such a backup is the most likely to contain all graphs.

If you add the name of the backup to be restored behind restore (skip the leading directory, e.g. rrd_backup_2016_05_26_13_40_03.tgz), the system will restore this backup.

If you need to restore data from system backups, section script, follow instructions in restore data from scripts backups to directory, copy this backup into a directory with backups for rrd ( /usr/local/script/ispadmin/rrd_data_backup/), and then follow instructions in this paragraph.

Increasing RAM disk manually

RAM disk is maintained until 80% of its capacity. If this capacity is reached, the system will try to increase RAM disk size by 20%.
You can define RAM disk size by running the following command:

/usr/local/script/ispadmin/rrd_ramdisk.pl set size_in_MB - Set size RamDisk to

Check and modify the settings of the following parameters:

• upload_max_filesize and post_max_size in /etc/php5/apache2/php.ini
• message_size_limit in /etc/postfix/main.cf

If the set maximum size is exceeded, the system will display an error message after you click on Send.

By default, the maximum email size is 10MB.

These are instructions on how to create and set up trustworthy certificates (https) for your ISPadmin installation and the Client portal. From the information in the address bar, you can find out whether the website in question is secured or not. Different browsers inform users of this fact in different ways. For example: If https appears in front of the URL/IP address of ISPadmin, then the website is secured. If https is crossed out, it is unsecured.

Fatal error: Allowed memory size of ... bytes exhausted (tried to allocate ... bytes) in ... on line ...

If such an error message appears, set a higher value in the memory_limit item in the PHP configuration file (/etc/php5/apache2/php.ini).

• headend
• ipaccount
• modem
• routers
• snmp
• switches
• user_devices
• wifi

If the Error 404: Not Found message appears during an update, it will be necessary to do it manually:

cd /tmp

# for beta version
wget http://www.ispadmin.cz/update/update_ispadmin_ion_beta_AddHashHere.tgz

# for stable version
wget http://www.ispadmin.cz/update/update_ispadmin_ion_AddHashHere.tgz

tar zxfv DownloadedFile.tgz

cd ispadmin_update

./update.pl