Monday, May 14, 2018

NTP Time Synchronization―All Things Considered

In this article, we will cover the following topics:
  1. Importance of time synchronization
  2. Needs for NTP time synchronization
  3. How to synchronize time
  4. How to verify synchronized time

Importance of Time Synchronization


Time synchronization plays an important role in cloud computing:
As single systems have been replaced by multiple, loosely-coupled systems, this need has evolved into a requirement for both accurate and consistent clocks among these systems. The need for millisecond or microsecond timestamp accuracy derives from the need to execute transactions in a correct sequence, particularly if many transactions occur almost simultaneously. 
You can read [1] for more details.

Needs for NTP Time Synchronization


Each of your network of servers has its own clock.  Computer clocks are notorious for drifting. They are typically based on inexpensive oscillator circuits or battery backed quartz crystals that can easily drift seconds and minutes per day, accumulating significant errors over time. That is why most enterprise networks today rely on network time servers that acquire time from the Global Navigation Satellite System (GNSS) and distribute it to clients over a network through the Network Time Protocol (NTP).

Timing accuracy from NTP servers depends upon the accuracy of the source and the precision with which all operations and applications are synchronized.

NTP Time Synchronization via utpupdate


ntpdate can be run manually as necessary to set the host clock, or it can be run from the host startup script to set the clock at boot time.

This is useful in some cases to set the clock initially before starting the NTP daemon ntpd. It is also possible to run ntpdate from a cron script. However, it is important to note that ntpdate with contrived cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use. Finally, since ntpdate does not discipline the host clock frequency as does ntpd, the accuracy using ntpdate is limited.

# /usr/sbin/ntpdate -u 10.252.148.42
14 May 09:57:05 ntpdate[2890]: step time server 10.252.148.42 offset 30.442100 sec

NTP Time Synchronization via NTP Daemon


Depending on your Linux systems, for example, below commands are used to update NTP server configuration file and restart ntpd in a local Oracle Linux system.


$ sudo bash
Password:

bash-3.2# vi /etc/ntp.conf
bash-3.2# /etc/init.d/ntpd restart
Shutting down ntpd:                                        [  OK  ]
ntpd: Synchronizing with time server:                      [  OK  ]
Starting ntpd:                                             [  OK  ]

Verification of Synchronized Time


There are multiple ways of verifying the results of NTP time synchronization:
  • ntpstat
    • ntpstat will report the synchronisation state of the NTP daemon running on the local machine.  
    • If the local system is found to be  synchronised to  a  reference  time source, ntpstat will report the approximate time accuracy.
  • date -R
    •  -R, --rfc-2822
      • output date and time in RFC 2822 format
  • /usr/sbin/ntpq -p
    • The  ntpq utility program is used to monitor NTP daemon ntpd operations and determine performance.
    • -p 
      • Print a list of the peers known to the server as well as a summary of their state. This is equivalent to the  peers  interactive command.

ntpstat

bash-3.2# ntpstat
unsynchronised
  time server re-starting
   polling server every 64 s


bash-3.2# ntpstat
synchronised to NTP server (10.252.148.42) at stratum 4
   time correct to within 31 ms
   polling server every 64 s

date

bash-4.1# date -R
Mon, 14 May 2018 08:17:33 -0700

/usr/sbin/ntpq -p

bash-4.1# /usr/sbin/ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ucf-c2z3-opc-di 10.68.0.42       3 u   20   64  377    0.971    0.105   0.057
 LOCAL(0)        .LOCL.          10 l    -   64    0    0.000    0.000   0.000


VariableDescription
remotehost name or IP number of peer
refidassociation ID or kiss code
stpeer status word
tu: unicast, b: broadcast, l: local
whensec/min/hr since last received packet
pollpoll interval (log2 s)
reachreach shift register (octal)
delayroundtrip delay
offsetoffset
jitterjitter


References

  1. The Importance of Network Time Synchronization for Enterprise Solutions Whitepaper
  2. Network Time Protocol (Version 3) Specification, Implementation and Analysis
  3. ntpq - standard NTP query program
  4. NTP Time Synchronization

Saturday, May 12, 2018

End-to-End Security―How to Debug SSL/TLS Issues

SSL (Secure Socket Layer) or its successor TLS (Transport Layer Security) are protocols to facilitate end-to-end security. These protocols are used when accessing web sites (https), delivering or retrieving email, and in lots of other use cases. In the following documentation we will refer to both SSL and TLS as simply 'SSL'.

When you debug SSL issues, you can use Linux curl command to test.  In this article, we will cover the following topics:
  • Curl and its backend OpenSSL
  • Which CA certificate was used
  • Unknown SSL protocol error 
  • Useful Tools for SSL Debugging

Curl and Its Backend OpenSSL


Depending on which working SSL library that your curl was built with, curl command may sometimes fail to negotiate SSL handshake because of older version of OpenSSL.  So, the first thing to check is which OpenSSL your curl command is built with by typing:

$ curl --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.51.0 OpenSSL/1.0.2j zlib/1.2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL libz

In this curl, it is built with OpenSSL 1.0.2j.

Specify CA Certificate with --cacert  Option


You can specify which CA certificate to verify peer against by using
 --cacert  
$ curl -v --cacert /u01/data/secure_artifacts/ssl/certs/cert-8b3a107e0d914aed91828f4414ad69f0_USPSRDEVCERTSIGN.pem https://10.240.81.24:3651
* Rebuilt URL to: https://10.240.81.24:3651/
* Trying 10.240.81.24...
* TCP_NODELAY set
* Connected to 10.240.81.24 (10.240.81.24) port 3651 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /u01/data/secure_artifacts/ssl/certs/cert-8b3a107e0d914aed91828f4414ad69f0_USPSRDEVCERTSIGN.pem
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 10.240.81.24:3651
* Curl_http_done: called premature == 1
* Closing connection 0

Default Certificate


 If you don't specify a CA certificate, the default certificate is used.  Notice that the default certificate used below is:
/etc/pki/tls/certs/ca-bundle.crt
$ curl -v  https://10.240.81.24:3651
* Rebuilt URL to: https://10.240.81.24:3651/
* Trying 10.240.81.24...
* TCP_NODELAY set
* Connected to 10.240.81.24 (10.240.81.24) port 3651 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 10.240.81.24:3651
* Curl_http_done: called premature == 1
* Closing connection 0

Unknown SSL protocol


From both curl commands' output, we have found the following error:
* Unknown SSL protocol error in connection to 10.240.81.24:3651
* Curl_http_done: called premature == 1
The reason turns out to be that listen port 3651 on server 10.240.81.24 is non-SSL.

Useful Tools for SSL Debugging


Often an error message alone is not sufficient to solve the problem. In this case the following tools can be of help:[2]
  • SSLLabs can be used to check problems with public accessible HTTPS servers. It shows problems about certificate verification and also about potential problems with specific SSL clients.
  • In case it is not https or the server is not public accessible analyze.pl from my SSL tools can help. It can be used to debug SSL problems with plain SSL or explicit SSL on SMTP, IMAP, POP3 and FTPS and with HTTP proxies.
  • openssl helps with debugging too, especially with the s_client, s_server and x509 commands.
  • And wireshark can be used to analyse packet captures done by tcpdump or wireshark. It is able to show lots of details about the SSL handshake.
For example, below outputs are from analyze.pl:

D:\Todo>perl analyze-ssl.pl  --all-ciphers -v3 10.240.24.81:3651
+ checking host=10.240.24.81(10.240.24.81) port=3651
* version SSLv23, no verification, ciphers= -> FAIL! SSL wants a read first
* version SSLv23, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1_2, no verification, ciphers= -> FAIL! SSL wants a read first
* version TLSv1_2, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1_1, no verification, ciphers= -> FAIL! SSL wants a read first
* version TLSv1_1, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version TLSv1, no verification, ciphers= -> FAIL! SSL wants a read first
* version TLSv1, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
* version SSLv3, no verification, ciphers= -> FAIL! SSL wants a read first
* version SSLv3, no verification, ciphers=HIGH:ALL -> FAIL! SSL wants a read first
10.240.24.81 failed basic SSL connect: SSL wants a read first

Note that the SSL_ERROR_WANT_READ means SSL handshake between client and server didn't complete.  In our case, this is because listen port 3651 is non-SSL.