Linux: How to Read Large Text File—/var/log/messages

To support Cloud Services, IaaS is the hardware and software that powers it all – servers, storage, networks, operating systems. These days Linux (or Windows) servers used in IaaS are more and more powerful. Hence they also generate more log files.

Very often we will run into large message files above 1 GB. These log files can be viewed by regular text editors. However, most text editors have a limitation of supporting files over a certain size.

In this article, we will cover how to read large message files (e.g., /var/log/messages) generated on Linux systems.

/var/log/messages

To debug issues in Cloud environments, it's essential for you to know where the log files are and what is contained in each log file. On Linux servers, over a dozen log files are located in /var/log directory. Here we only focus on one of them:

• /var/log/messages^[7]
• This log aims at storing "general system activity" messages.
• There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.
• The severity of messages could be
• [INFO]
• [DEBUG]
• [WARNING]
• [ERR]
• etc
• Older message files are archived periodically with their name annotated with the date.

If your Linux system uses rsyslogd utility, its configuration file is

/etc/rsyslog.conf

in which you can specify rules (i.e., selector + action) of logging. For example, you can log anything of level informational or higher except mail, cron, or private authentication message:

*.info;mail.none;authpriv.none;cron.none /var/log/messages

and messages are logged into a file named /var/log/messages.

Limitations of Text Editors

Some editors have limitations of supporting certain sizes of text file. For example, the following popular editors on Windows have described limitation:

• Notepad^[3]
• 64 kilobytes (KB)
• Wordpad^[4]
• It's said of no size limit. But, the real problem is performance.
• Depends on the version of Wordpad, some people say it can support files of size up to 20 MB without performance issues.
• Textpad^[8]
• It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory.

Solutions

Basically, there are two solutions of dealing with large text files:

1. Find a more capable text editor
2. Divide and conquer

If you google search "large text file", you may find many suggestions on Large Text File Reader. Some editors may be able to open and read large text files. However, the performance (e.g., searching a pattern) of it could be slow.

On Linux systems, a good approach is 'divide-and-conquer" by using split command like:

split -b1000m messages-20160315T2201 split-messages

After splitting, a good text editor such as Textpad will be able to read a file of 1000 MB easily.

References

1. 20 Linux Log Files that are Located under /var/log Directory
2. How to split large file into several smaller files – Linux
3. "File Is Too Large to Open" Message in Notepad
4. Wordpad - File size Limit ?
5. Text editor to open big (giant, huge, large) text files
6. Configuring and auditing Linux systems with Audit daemon
7. Difference between /var/log/messages, /var/log/syslog, and /var/log/kern.log?
8. Textpad specification