Thursday, June 26, 2014

TextPad: How to Remove All Lines Except the Ones Containing a Pattern

Similar to NotePad,[1] you can remove all lines except the ones containing a pattern in TextPad.[2,3] There is an easy way to achieve this. Basically, you need to perform 3 steps:
  1. Bookmark all lines containing a pattern
  2. Inverse bookmark
  3. Delete all bookmarked lines

 

Bookmark All Lines Containing a Pattern


For this demonstration, we want to remove all lines except the ones containing the following pattern:
  • "concurrent-mark-end"
To bookmark all lines containing a pattern:
  1. Go to Menu "Search" and select "Find..."  
  2. Specify the pattern in "Find what:" and select "Regular expression"
  3. Hit Button "Mark All"

Inverse Bookmark


To inverse bookmark, do:
  • Go to Menu "Search" and select "Inverse Bookmarks"
 You should see some lines are marked and some not.


Delete All Bookmarked Lines


To delete all bookmarked lines, do:
  1.  Go to Menu "Edit" and select "Delete > Bookmarked Lines"
and you should see the final results below:



References

  1. How to delete all line except lines containing a word I need?
  2. TextPad Quick Reference Card - John Bokma
  3. TextPad Downloads

Wednesday, June 25, 2014

Oracle Fusion Apps: The port may already be used by another process

After cloning our benchmark (i.e., CRM FUSE) to new servers, the following issue popped up:
The port may already be used by another process
This happened only after I restarted some managed servers in the WebLogic Domain.  In this article, we will describe what happened and how to fix it.

The Issue


The port in conflict was 9020.  As described in a companion article[1], you can use netstat command in Linux to investigate:
$ netstat -an |grep 9020

tcp  0  0 ::ffff:10.214.10.20:7101    ::ffff:10.214.10.20:9020    ESTABLISHED 
tcp  0  0 ::ffff:10.214.10.20:9020    ::ffff:10.214.10.20:7101    ESTABLISHED 

So, port 9020 was truly used by another process.  What happened is that we had many Fusion applications running on the same server and many sockets were created.  For a specific application, it requires to use port 9020 to listen and this port happened to be grabbed by another process dynamically.

In [1], we have documented a way to walk around port-conflict issue—by re-ordering start-up steps.  But, that cannot be guaranteed to work every time.  So, we will look at another approach in this article.

TCP Socket


It is the socket pair that specifies the two endpoints that uniquely identifies each TCP connection in an internet.  Note that an internet connection can use different transport protocols.  Here we only cover TCP connection.

For either local or remote endpoint, it is a combination of an IP address and a port number, much like one end of a telephone connection is the combination of a phone number and a particular extension. Based on this address, TCP sockets deliver incoming data packets to the appropriate application process or thread.

A process that opens a listen port will allow multiple sockets to that port. For example, when tnslsnr listens on port 1521, there will be many sockets where one port is 1521. So that port is shared; it can only be used for connections to that one particular process. The OS will never pick that port for the dynamic side of a connection, and any attempt by another process to listen on that port will fail because the address is in use.

The other port (picked by the OS) can not be shared; it will be used exclusively by the socket assigned to that port.  For the dynamic port, it will be picked up from the ip_local_port_range. For example, on our Linux server,  it was set to be from 9000 to 65500
$cat /proc/sys/net/ipv4/ip_local_port_range
9000    65500

The Solution


Port-conflict happened when a connection tried to pick  an available port (i.e., 9020) from the range (i.e., from 9000 to 65500).  In our case, it has picked 9020, which happened to be required by another managed server.

To prevent port conflicts from happening, we need to raise the lower limit of ip_local_port_range to be higher (say, 11000):
# echo "11000 65500" >/proc/sys/net/ipv4/ip_local_port_range

Note that you need to be the root user to make this change.  If you use Redhat edition, read [2] for more details.

As you can tell, after the ip_local_port_range change, the system tried to pick dynamic ports from 11000-to-65500 range when it connects to a remote socket (note that this remote endpoint happens to be on the same server):
$ netstat -an |grep 7101

tcp  0  0 ::ffff:10.214.10.20:7101   :::*                        LISTEN
tcp  0  0 ::ffff:10.214.10.20:7101   ::ffff:10.214.10.20:19739   ESTABLISHED
tcp  0  0 ::ffff:10.214.10.20:7101   ::ffff:10.214.10.20:20506   ESTABLISHED

Before we end this article, we also want to share two nice-to-know topics:
  • Privileged Ports
  • Which Port Is Configured for AdminServer or Managed Servers

Privileged Ports


The port numbers are divided into three ranges:
  • Well Known Ports: those from 0 through 1023.
  • Registered Ports: those from 1024 through 49151
  • Dynamic and/or Private Ports: those from 49152 through 65535
The TCP/IP port numbers below 1024 are special in that normal users are not allowed to run servers on them. This is a security feature, in that if you connect to a service on one of these ports you can be fairly sure that you have the real thing, and not a fake which some hacker has put up for you.

When you run a server as a test from a non-privileged account, you will normally test it on other ports, such as 2784, 5000, 8001 or 8080, not the well-known port (say, 80).

Which Port Is Configured for AdminServer or Managed Servers


If you are not sure about the port used by Weblogic server's Admin and Managed servers, you can verify it from the configuration file $DOMAIN_HOME/config/config.xml.   For example, port 9020 was reserved for our CRMAnalyticsServer_1:

<machine>slcaf977.us.oracle.com</machine>
    <listen-port>9020</listen-port>
    <cluster>CRMAnalyticsCluster</cluster>
    <web-server>
      <name>CRMAnalyticsServer_1</name>

References

  1. How to Investigate: Failed to Bind to Port on Linux
  2. The ip_local_port_range parameters (Redhat edition)
  3. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)

Tuesday, June 24, 2014

HotSpot: What Are Those GC Worker Threads in G1 GC?

On the Internet, you can find some tuning guides for Garbage First Garbage Collector (G1 GC)[1,5,8,9] and some of them may be outdated.

In this article, we try to update some of those information with the focus on thread tuning.

Ergonomics


Ergonomics for servers was first introduced in Java SE 5.0[2]. It has greatly reduced application tuning time for server applications, particularly with heap sizing and advanced GC tuning. In many cases no tuning options when running on a server is the best tuning you can do.

Here we will focus on the number of GC worker threads chosen to run on a specific server based on the ergonomics in G1 GC.

Garbage Collector
Worker Threads Used
Parallel
ParallelGCThreads
CMS
ParallelGCThreads
ConcGCThreads
G1
ParallelGCThreads
ConcGCThreads G1ConcRefinementThreads


G1 Threads


In the table below, we have summarized the most important GC worker threads in Oracle's G1 GC implementation. For comparison, Parallel GC only uses ParallelGCThreads and CMS only uses ParallelGCThreads and ConcGCThreads. Be warned that these values can be changed in the future if Oracle sees appropriate.

GC thread counts are based on the number of processors (including virtual processor if hyper-threading[3] is enabled) reported by the system. To see the number of processors you have in the system, check out the file /proc/cpuinfo. In this article, 32 processors are used in the demonstration.



NameControlled byDefault if Not SpecifiedDescription
Parallel GC Threads-XX:ParallelGCThreadsif #proc <=8, then #proc;
if #proc >8, 8+(#proc-8)*(5/8)
Parallel operations
Parallel Marking Threads-XX:ConcGCThreadsThere are lot of cases, but mostly it uses max((ParallelGCThreads+2)/4, 1)
G1 Main Concurrent Mark GC Thread
1Master concurrent thread
G1 Concurrent Refinement Thread-XX:G1ConcRefinementThreadsParallelGCThreads+1The +1 is the master thread which control these worker threads.


What Are They?


G1 garbage collection cycle has multiple phases: it starts with an initial marking phase and ends with a concurrent-cleanup phase. During the cycle, there are some phases that pause all the application threads, and some that run concurrently. These GC worker threads used in G1 GC include, but not limited to:
  • ParallelGCThreads
    • ParallelGCThreads are the threads employed during GC pauses
      • They are responsible for copying (and scanning for) live objects during stop-the-world (or STW) pauses.
      • They are also employed during other STW pauses (such as Remark and Cleanup).
  • ConcGCThreads
    • In concurrent marking, there is one controller thread and another bunch of worker threads. The number of worker threads is set using ConcGCThreads or a fraction of ParallelGCThreads if ConcGCThreads is not set.
      • Worker threads are the threads used for marking and determining the liveness of regions, which are run concurrently to the application threads.
      • Increasing ConcGCThreads will make the marking cycle complete faster.
  • G1ConcRefinementThreads
    • G1ConcRefinementThreads are the threads used for updating remembered set while application threads are running concurrently.
      • They are responsible for taking the buffers that are used to log object updates and updating the RememberedSets (or RSets)[10] based upon the updates.
    • If G1ConcRefinementThreads is not set, then its value is ParallelGCThreads + 1.
      • For example, if ParallelGCThreads is 23 then we will have 24 refinement threads.
        • One is used for sampling and updating the RSets of young regions (typically young regions have less cross-region references so one thread is enough); the other 23 threads fill update buffers.
        • Typically they don't all run at once.
          • They each have a stepped activation threshold and a thread is activated when the number of completed buffers exceeds it's activation threshold.
          • They also have deactivation thresholds and when the number of filled update buffers falls below a thread's deactivation threshold, it is deactivated.

Thread Tuning


For our server with 32 processors, the number of GC worker threads chosen by G1's ergonomics are:
  • # ParallelGCThreads = 23
  • # ConcGCThreads = 6
  • # G1ConcRefinementThreads = 23

To find out what the GC worker thread counts are for your server, you can use -XX:+PrintFlagsFinal.[4] You can also change the default values if you see appropriate.[6,7] For some systems, default values sometimes can be too high.

References

  1. Garbage First Garbage Collector Tuning (Published August 2013)
  2. Java Heap Sizing: How do I size my Java heap correctly?
  3. Intel® Hyper-Threading Technology
  4. What Are the Default HotSpot JVM Values?
  5. Part #1 - Tuning Java Garbage Collection for HBase
  6. Linux: Understanding Processor Queue in Vmstat Output
  7. How to Troubleshoot High CPU Usage of Java Applications? 
  8. Getting Started with the G1 Garbage Collector 
  9. G1: One Garbage Collector To Rule Them All 
  10. Remembered Sets (RSets) in G1 GC
    • G1 GC uses independent Remembered Sets (RSets) to track references into regions. Independent RSets enable parallel and independent collection of regions because only a region's RSet must be scanned for references into that region, instead of the whole heap. To achieve the task, G1 GC uses a post-write barrier to record changes to the heap and update the RSets. 
  11. D. L. Detlefs, C. H. Flood, S. Heller, and T. Printezis. Garbage-First Garbage Collection. In A. Diwan, editor, Proceedings of the 2004 International Symposium on Memory Management (ISMM 2004), pages 37-48, Vancouver, Canada. October 2004. ACM Press.
  12. T. Printezis and D. L. Detlefs. A Generational Mostly-Concurrent Garbage Collector. In A. L. Hosking, editor, Proceedings of the 2000 International Symposium on Memory Management (ISMM 2000), pages 134-154,Minneapolis, MN, USA, October 2000. ACM Press. 
  13. g1gc logs - how to print and how to understand - basic 
  14. G1 GC Glossary of Terms
  15. Garbage-First Garbage Collector (JDK 8 HotSpot Virtual Machine Garbage Collection Tuning Guide)

Wednesday, June 18, 2014

Cloning: Listener won't start after cloning

This article is a companion article to our previous cloning series:
As pointed out in the previous articles, there are limitations and issues with cloning (either of an application or a database). This article describes one of the new issues.

TNS-00525: Insufficient privilege for operation


After cloning our OATS server to a new server, we have seen the following listener error when we restarted OATS' database on the new server:

### Starting tns listener ###

LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 10-JUN-2014 19:48:43
Copyright (c) 1991, 2010, Oracle.  All rights reserved.
Starting /scratch/aime1/app/aime1/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.2.0 - Production

System parameter file is /scratch/aime1/app/aime1/product/11.2.0/dbhome_1/network/admin/listener.ora

Log messages written to /scratch/aime1/app/aime1/diag/tnslsnr/myOatsServer/listener/alert/log.xml

Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
TNS-12555: TNS:permission denied
 TNS-12560: TNS:protocol adapter error
  TNS-00525: Insufficient privilege for operation
   Linux Error: 1: Operation not permitted

The Solution


After trying tool such as truss or strace on Linux, it didn't help much.  Then, I have found articles [3,4] which have helped me solve the issue:

bash-3.2$cd /var/tmp

bash-3.2$ ls -la .oracle
total 12
drwxrwxrwt 2 root   root 4096 Mar 17 11:38 .
drwxrwxrwt 3 root   root 4096 Jun 10 18:28 ..
srwxrwxrwx 1 oracle lock    0 Mar 17 11:38 s#16150.1
srwxrwxrwx 1 oracle lock    0 Mar 17 11:38 s#16150.2
srwxrwxrwx 1 oracle lock    0 Feb 24 10:17 s#32488.1
srwxrwxrwx 1 oracle lock    0 Feb 24 10:17 s#32488.2
srwxrwxrwx 1 oracle lock    0 Mar 17 11:38 sEXTPROC1521

As shown above, our new server has an existing folder named .oracle under /var/tmp, which was owned by a different user (i.e. oracle).  After we have backed up our original .oracle folder, we have changed the owner of /var/tmp/.oracle to be aime1 and removed the files within it.  After restarting our listener, here are the new files generated in it:

.oracle:

total 0
srwxrwxrwx 1 aime1 svrtech 0 Jun 18 07:50 s#30309.1
srwxrwxrwx 1 aime1 svrtech 0 Jun 18 07:50 s#30309.2
srwxrwxrwx 1 aime1 svrtech 0 Jun 18 07:50 sEXTPROC1521


References

  1. truss(1) equivalent in Linux ?
  2. 5 simple ways to troubleshoot using Strace
  3. Can't start listener for XE on Redhat Enterprise Linux 5.3 
  4. Listener not starting with TNS-00525: Insufficient privilege for operation
  5. Oracle Products: What Patching, Migration, and Upgrade Mean? (Xml and More)