Saturday, March 19, 2016

Linux: How to Read Large Text File—/var/log/messages

To support Cloud Services, IaaS is the hardware and software that powers it all – servers, storage, networks, operating systems. These days Linux (or Windows) servers used in IaaS are more and more powerful. Hence they also generate more log files.

Very often we will run into large message files above 1 GB. These log files can be viewed by regular text editors. However, most text editors have a limitation of supporting files over a certain size.

In this article, we will cover how to read large message files (e.g., /var/log/messages) generated on Linux systems.

/var/log/messages


To debug issues in Cloud environments, it's essential for you to know where the log files are and what is contained in each log file. On Linux servers, over a dozen log files are located in /var/log directory. Here we only focus on one of them:
  • /var/log/messages[7]
    • This log aims at storing "general system activity" messages.
      • There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.
      • The severity of messages could be
        • [INFO]
        • [DEBUG]
        • [WARNING]
        • [ERR]
        • etc
    • Older message files are archived periodically with their name annotated with the date.
If your Linux system uses rsyslogd utility, its configuration file is
/etc/rsyslog.conf
in which you can specify rules (i.e., selector + action) of logging. For example, you can log anything of level informational or higher except mail, cron, or private authentication message:
*.info;mail.none;authpriv.none;cron.none /var/log/messages
and messages are logged into a file named /var/log/messages.

Limitations of Text Editors


Some editors have limitations of supporting certain sizes of text file. For example, the following popular editors on Windows have described limitation:
  • Notepad[3]
    • 64 kilobytes (KB)
  • Wordpad[4]
    • It's said of no size limit. But, the real problem is performance.
    • Depends on the version of Wordpad, some people say it can support files of size up to 20 MB without performance issues.
  • Textpad[8]
    • It can handle file sizes up to the largest contiguous chunk of 32-bit virtual memory.

Solutions


Basically, there are two solutions of dealing with large text files:
  1. Find a more capable text editor
  2. Divide and conquer
If you google search "large text file", you may find many suggestions on Large Text File Reader. Some editors may be able to open and read large text files. However, the performance (e.g., searching a pattern) of it could be slow.

On Linux systems, a good approach is 'divide-and-conquer" by using split command like:
split -b1000m messages-20160315T2201 split-messages

1 comment:

Blogger said...

Submit your website or blog now for appearing in Google and 300+ search engines!

Over 200,000 websites submitted!

Submit TODAY via I NEED HITS!!!