Linux System Administration Best Practices

This guide will describe the decisions and tasks involved in running a secure Linux or unix-based system. It does not contain step by step instructions, because these vary between systems. You can use an Internet search engine to find detailed instructions. If you are affiliated with SEAS, you can contact CETS for advice, but we are not familiar with every operating system, distribution, and version.

It is best if each machine has one person that is responsible for managing and maintaining the machine. Other people can help out, but if no one is in charge of making sure that the machine is patched and backed up, then often everyone assumes that someone else will do it.

You have to worry about backups and security for every OS installation, which includes dual boot and virtual installations. It is very difficult to keep a machine patched if it is the "unbooted" side of a dual-boot machine, so when it is booted it is often immediately compromised. We don't support dual boot systems for this reason - we simply can't keep them secure.

Backups

If you are responsible for a machine, you are responsible for the files stored on that machine. You must decide how to back up the files. If you are taking responsibility for a system that someone else set up, you should review the backup decisions to make sure that you understand them, and that they are still appropriate. A yearly test restore and review of your backup model is a good idea.

Backups allow you to recover if your machine is compromised, or if there is a disk failure or other hardware failure. Disks often fail suddenly and without warning, and events such as fires, electrical surges, and theft can cause data loss with no warning. Mirrors or RAIDs do not protect data from fire, electrical surges, physical theft, or compromised security (cyber break-ins).

In the past, people in SEAS have lost many months of work because they did not have a reliable backup. This sort of disaster can result in losing valuable research data, papers ready to submit for publication, grant proposals in preparation, etc. Many grants require that the computers involved in the research be administered according to currently accepted "best practices" which include regular backups and secure configuration.

There are five basic approaches to backing up your machine:

1. Store important files on the SEAS file server, and only store temporary files, the OS and applications on your machine. If necessary, the OS and applications can be reinstalled from original media or from a trusted source.
2. Store important files on your local disks, and contract with CETS to perform automatic network backups (see the "Help from CETS" section below).
3. Store non-volatile data (eg, input data) away from the machine on DVD or other media, and have a working copy on the machine's disks.
4. Have two machines, preferably in different rooms or buildings, and automatically copy data from one to the other. This doesn't help if the deleted or corrupted data is automatically copied before you notice the problem. Ideally you would have more than one remote copy, perhaps a daily copy and a weekly copy.
5. Copy your data periodically to a DVD or other media and store the copies away from the machine. We don't recommend this method. In our experience, when something goes wrong and you need the backups, you discover that the person who was supposed to do the backups graduated and your most recent backup is six months old.

It's not always possible to back up a database directly. The database files may not be in an internally consistent state or data may be stored in memory. Dump a snapshot of the database to flat files and then back them up, or halt the database server, perform backups, and restart the database server. See your database documentation for how to do backups.

Security

We are seeing constant, aggressive attacks on SEAS machines. The attackers are methodically scanning the network, so no machine is safe. They have already gotten root on four SEAS machines that we know of.

We have seen machines where every disk was wiped clean by someone who had broken into the machine. Taking back control of a compromised machine is time consuming, which can be disastrous if there is a deadline approaching. The machine will be unavailable for a time, and you and the other users of the system will need to spend time securing your passwords and accounts.

Most of the compromises that we have detected have come in via SSH. We have also seen successful attacks come against web applications. There is no single remedy for these attacks, short of disconnecting your machine from the network.

SSH

Consider turning off incoming SSH connections completely. SSH is used for remote shell access and remote execution of commands, and also for tunneling other protocols such as FTP and X Windows. SSH, like all network services, should be disabled if not needed.

If you need SSH, we recommend disabling root logins via SSH. Root is the account that they try the most and that you want to protect the most, so block all SSH logins to root. To work remotely as root, login as a normal user and su to root. In most cases, you can just add this line to /etc/ssh/sshd_config .

PermitRootLogin no

Lock out connections from IPs that have been trying bad passwords. Leave the IPs blocked forever. The SSH parameter MaxAuthTries limits the number of failed passwords per connection, but does not permanently block the IP address. The attacker can just connect back. Intrusion detection systems like OSSEC and fail2ban can monitor your system for failed SSH connections and block attackers automatically.

Running the SSH daemon on a different port can reduce the number of attacks that you will receive.

Firewall

Make sure that your system has a firewall. There are different ways of implementing firewalls. For most Linux distros, iptables is a good choice. Ubuntu ships with ufw by default. firewalld is the default on Red Hat Enterprise Linux and SUSE based distributions, and is available to install on most others.

If possible, do not allow connections from unknown networks. Use iptables or similar firewall software to allow only hosts or networks that you and the other users use. When traveling, people can SSH to eniac.seas.upenn.edu, and SSH from there to your machine.

The more tightly you can restrict network access, the better. It's better to allow only machines at Penn, Stanford, and CMU, than to leave it open to the entire Internet. Until we have a better sense of how IPv6 impacts system security, we recommend blocking all IPv6 incoming connections unless you have a specific need for IPv6.

Block all unnecessary services on your machine. You should be aware of all network services offered by your computer, and you should know how each is used and why it is necessary. FTP, X Windows, rsync, etc should always be tunneled through SSH. You should not have ports open for those services. Ask CETS to scan your machine and tell you what ports are open on your machine.

Patches

If possible, configure your software to automatically install security patches. Security patches are often released in response to attacks that are actively being used throughout the Internet.

It is a good idea to install security patches the day that they are announced. Unless you have a team of sysadmins or you want to install patches when on vacation or over break, it's best to install them automatically.

Attackers know that they have a small window of time before machines are patched, so they try to attack as many machines as possible as quickly as possible. Often these attacks are made on a holiday when the attackers hope that most sysadmins will not be able to patch their machines. These attacks are automated, with each newly compromised machine joining in the attacking force.

Any application can have a security hole. For example, Oracle, Adobe Reader, Adobe Flash, and Java have had serious security holes that needed patching. Install any application patches the day that they come out, same as OS patches.

Accounts

Set a root password that is at least ten characters and unusual. Only use the root account for systems recovery. Login as yourself and use sudo when you are doing systems administration.

At least once a year, you should remove or lock accounts that are no longer needed, and make sure that the passwords on active accounts are changed. If you can't get in touch with someone to discuss their account and password, then we recommend removing or locking the account. If you use email, make sure that you are not exchanging email with someone who has cracked the email account. It's much better to discuss account security by phone or in person.

You should have different passwords on machines managed by different people or organizations. Confidential data should be protected by different passwords or additional passwords. For example, use different passwords for your PennKey password, a machine you run yourself, online services, etc.

Other Suggestions

Email

Don't run a mail server. If you run a mail server you not only have to worry about compromises to your machine through that service, but you need to worry about preventing spammers and others from using your mail server to send spam and phishing email. Running a mail server securely is a huge pain, so just use the SEAS mail server, gmail, hotmail, etc.

Mail sent to every account on your machine should be forwarded to wherever that person reads mail, and mail sent to any system account (eg "root") should be forwarded to wherever you read mail. Many software services or systems send mail to "root" if something unusual happens. Often break-ins are detected because of some small side-effect which triggers a warning. For example, someone breaks in and fills up a partition with large amounts of copyrighted material. That causes some routine accounting task to fail because it can't create a temporary file. You (root) get mail that a routine accounting task failed, investigate, and discover that the machine has been compromised.

Servers

If you are running a server, contact CETS. We can give you a static IP address and hostname. Web access is blocked to all SEASnet machines by default. If you are running a web server, we can enable web access to your machine.

Keep informed

Subscribe to the security announcement mailing list for your operating system version and any applications that you have installed. These lists will announce when new security patches are available and may also describe workarounds to keep your system secure until a patch is released.

Consider taking a Linux systems administration course.

Subscribe to Security-SIG at to receive announcements of local issues and training opportunities.

Subscribe to alerts and advisories from the United States Cybersecurity and Infrastructure Security Agency (CISA).

Monitor your machine's logs and processes regularly, so that you can recognize what's normal and unusual on your machine. Make sure the Network Time Protocol (NTP) daemon is configured to run at boot time and is correctly synchronizing your machine's clock. This will ensure that the logs have accurate timestamps.

Prepare for the worst

Read this CERT article, and make sure that you have everything that you'd need to recover from a root compromise.

Make an archival backup of the operating system and applications in order to speed up restores in the future to a known clean state.

Help from CETS

Contact your computer support organization if you see evidence that a machine or an account has been compromised (this is required by Penn policy). If you are affiliated with SEAS, contact CETS. We can help you through the recovery process. We do not charge for providing advice to SEAS sysadmins.

CETS can provide some services for a fee, such as providing weekly, automated backups over our network. The cost of this service varies depending on the amount of data that you have and other factors. For current pricing please contact CETS. CETS can also do systems administration work on machines, such as configuring and securing the machine, setting up services, etc. There is an hourly rate charged for this sort of work. The rate depends on the amount of experience of the individual doing the work.