Cross Column

Showing posts with label Metadata. Show all posts
Showing posts with label Metadata. Show all posts

Thursday, August 14, 2014

HotSpot: Understanding Metaspace in JDK 8

In JDK 8, it will print out the following information when JVM exits:
Heap
 <snipped> 
 Metaspace       used 2425K, capacity 4498K, committed 4864K, reserved 1056768K
  class space    used 262K, capacity 386K, committed 512K, reserved 1048576K
if you have enabled GC printing using:
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Note that if PrintGCDetails is enabled, PrintGC[1] is also enabled.

In this article, we will examine what the above lines mean.

Metaspace


Internal JVM memory management is, to a large extent, kept off the Java heap and allocated natively in the operating system, through system calls like malloc or mmap. This non-heap system memory allocated by the JVM is referred to as native memory. Similar to the Oracle JRockit and IBM JVM's, the JDK 8 HotSpot JVM is now using native memory for the representation of class metadata, which is called Metaspace.[2,4]

In JDK 8, Hotspot explicitly manages the space used for metadata. Space is requested from the OS and then divided into chunks. A class loader will allocate space for metadata from its chunks (a chunk is to bound to a specific class loader). When classes are unloaded for a class loader, its chunks are recycled for reuse or returned to the OS.   Note that metadata uses mmap'ed space and not malloc'ed space.

Used/Capacity/Committed/Reserved


When JVM exits, it prints the following information if you have enabled GC printing:
 Metaspace       used 2425K, capacity 4498K, committed 4864K, reserved 1056768K
  class space    used 262K, capacity 386K, committed 512K, reserved 1048576K
In the “Metaspace” line, the “used” is the amount of space used for loaded classes and other metadata. The “capacity” is the space available for metadata in currently allocated chunks. The “committed” value is the amount of space available for chunks. The “reserved” is the amount of space reserved (but not necessarily committed) for metadata. On the “class space” line, those values are the corresponding values for the class area in Metaspace when compressed class pointers are used.

Mataspace is dynamically managed by HotSpot.  Metadata are deallocated when their class loader are garbage collected (GC'ed).  A high water mark is used for inducing a GC—when committed memory of all metaspaces reaches this level, a GC is triggered.  You can use the following flag:
-XX:MetaspaceSize=<size>
to specify the initial high water level. 

The interaction between metaspace growth and GC can be summarized as follows:
Metaspaces expand the native memory it is using until it gets to some level (starts at MetaspaceSize). When it hits that level, it does a GC to see if classes can be unloaded. After the GC, it can use freed space in it for metadata. If not enough space has been freed, it uses more native memory.
After the GC it also decides what the next level is for doing a GC to unload classes.  The level mostly increases to have fewer GC's. It sometimes will decrease if lots of space has been freed due to class unloading.  If MetaspaceSize is set higher, then fewer GC will be done early.

Acknowledgement


Some writings here are based on the feedback from Jon Masamitsu. However, the author would assume the full responsibility for the content himself.

References

    1. Java HotSpot VM Options
    2. HotSpot: Monitoring and Tuning Metaspace in JDK 8
    3. Latest JDK 8 and JDK 7 available here
    4. JavaOne Wednesday: Permanent Generation Removal Overview   
    5. jstat Tool: New Metaspace Statistics from -gc Option 
    6. Metatspace in JDK 8 (good) 
    7. VM Class Loading
    8. Learn More About Performance Improvements in JDK 8 

    Sunday, May 5, 2013

    Book Review: Managing Multimedia and Unstructured Data in the Oracle Database

    There is a large amount of unstructured data in the real world. In 1998, Merrill Lynch cited a rule of thumb that somewhere around 80-90% of all potentially usable business information may originate in unstructured form[1].

    There is no doubt that we need to process this information to extract meaning and create structured data about the information.  One approach will be using Oracle Database to manage such data which includes multimedia (aka "rich media").  This book:
    aims to help readers understand and manage unstructured data using Oracle Database.


    Dealing with Unstructured Data


    Unstructured Data refers to information that either does not have a pre-defined data model and/or does not fit well into relational tables[1].

    Relational data can be considered a subset of structured data.  Besides relational databases, structured data can also be stored in non-relational format such as in XML, inverted list databases[2], or object databases[4]. XML does not conform with formal structure of data models, but nonetheless it contains tags to separate  semantic elements and enforce hierarchies of records and fields within the data.  Therefore, one can consider XML as semi-structured.

    It's possible to store unstructured data in a column of a relational table, which is structured.  The traditional approach has been to just treat it as a blob (binary large object), but with a greater understanding of the variety of unstructured data types (i.e., video, audio, photographs, documents, etc.) that exist, the need to manage them has grown.

    Metadata


    To manage unstructured data, metadata is crucial. It is the data that describes the unstructured data and gives meaning to it.   Metadata can be used for
    • Searching (covered in Chapter 4, Searching the Multimedia Warehouse)
    • Annotation
      • Adding meaning to unstructured data objects
    • Relating unstructured data objects (or adding structure)
    • Matching data stored in relational databases
    It is envisaged that in the future technology will improve to the point that algorithms will be able to identify objects and people in a video or photo, and understand sounds and complex speech in audio files. When that point is reached, the need for metadata may be reduced or limited to a smaller scope.

    Oracle Database


    In the past few years, with changes in database technology and improvements in disk performance and storage, it now makes business sense to use the Oracle database to store and manage all of an organization's unstructured data.

    For a database management system to begin to correctly handle the unstructured data, it must have support for objects[4].  The use of a database that can support objects makes it a lot easier to manage large volumes of digital objects. Though these objects can be stored in a file system, there are now advantages to having them stored inside the database.  In Oracle, both relational data and objects are supported.  After adding Online Analytical Processing (OLAP)[5] and XML[6], Oracle database grew from being relational to one supporting most structures.

    Oracle multimedia uses blobs and new types, which can be accessed and used as required. In addition, it supports a variety of methods that simplify the act of loading and manipulating digital objects. This is covered in in Chapter 7, Techniques for Creating a Multimedia Database.

    Most databases can enable unstructured data to be stored in them, but do not support the management, control, and manipulation of that data. Even though Oracle is a market leader in unstructured data management there are still a large number of major improvements needed.  This is covered in Chapter 9, Understanding the Limitations of Oracle Products.

    Scalability


    When working with multimedia and unstructured data, a row in the database can be 10 GB in size, which could be greater than an entire relational database.  Therefore, traditional tuning techniques might fail as the rules regarding them no longer make sense.

    For example, in a multimedia warehouse (covered in Chapter 4), the concept of trying to achieve logical data consistency is not attempted, as it becomes apparent that the amount of data that is fuzzy forms the bulk of most of the digital objects.  So, novel solutions to tuning problems are needed.

    In Oracle Multimedia, there are also built-in supports for scalability.  For example, the new 11G Securefile BLOBS using parallel techniques, that allows of loading of files much faster than using traditional BLOBS.

    From the hardware front, technology also offers help.  With the recent introduction of a low-cost terabyte SATA disks, and with the use of low-cost SAN's, the ability to store a petabyte is within the reach of a number of organizations.

    To handle large amount of unstructured data, issues seen and solutions provided by Oracle include:

    • Hitting limits on the image size
      • An Oracle BLOB can be unlimited in size
    • Reaching internal structural limits within the database (max number of files that store data)
      • Oracle's use of tablespaces allow a large number of multimedia files to be stored in it
      • With the ability to control where a blob is stored, files can be split across multiple tablespaces and devices 
    • Dealing with fragmentation 
      • By using locally managed tablespaces, fragmentation is removed as a performance issue. 
    • The efficient management of those images (for example, backup/recovery)
      • Using partitioning on LOBS allows a very large number of multimedia files to be stored and efficiently managed
      • Using RMAN, Oracle can be configured to back up large amounts of data
    When dealing with Multimedia, one has to look at the different dimensions of scalability to best understand how the Oracle Database best handles it.  This includes managing the CPU, memory, disk I/O, and network bandwidth. All of the above are covered in Chap 8, Tuning.

    References

    1. Unstructured data (Wikipedia)
    2. ADABAS (Wikipedia)
    3. 3D Printing (Wikipedia)
    4. Object Database (Wikipedia)
    5. Oracle OLAP
    6. Oracle XML DB
    7. Managing Multimedia and Unstructured Data in the Oracle Database (Reviewed Book)

    © Travel for Life Guide. All Rights Reserved.

    Analytical Insights on Health, Culture, and Security.