OCI―Knowing File Storage Service Basics

Video 1.  OCI Level 100 - File Storage (Demo starts at 23:00)

Oracle Cloud Infrastructure (OCI) File Storage Service (FSS) is a managed file storage service that

• Provides a durable, scalable, distributed, enterprise-grade network file system.
• Using industry-standard Network File System (NFSv3) file access protocols and file system semantics
• You can start with a file system that contains only a few kilobytes (KB) of data and scale to exabytes (EB) of data without upfront provisioning. 
• It protects your data by maintaining multiple replicas, providing transparent encryption, and enabling frequent snapshots.
• Can be accessed concurrently by thousands of compute instances
• You can connect to a FSS file system from any bare metal, virtual machine, or container instance in your Virtual Cloud Network (VCN)
• You can also access a file system from outside the VCN using Oracle Cloud Infrastructure FastConnect and Internet Protocol security (IPSec) virtual private network (VPN)

FSS Security

You create users and groups in OCI. Then, you can use policies to specify which users and groups can create, access, or modify resources such as file systems, mount targets, and export options.

There are four distinct and separate layers of security to consider when using the FSS. Each layer has its own authorization entities and methods which are separate from the other layers.

This security layer...	Uses these...	To control actions like...
IAM Service	OCI Users and policies	Creating instances (NFS clients) and VCNs. Creating, listing, and associating file systems and mount targets.
Network Security	IP addresses, CIDR blocks, security lists	Connecting the client instance to the mount target. This is on an ‘all or nothing’ basis - the client either can or cannot access the mount target, and therefore all file systems associated with it.
NFS export options	File system exports, IP addresses, Unix users	Applying access control per-file system based on source IP CIDR blocks that bridges the Security Lists layer and the NFS v.3 Unix Security layer. See Working with NFS Export Options to specify granular controls on a per-file system basis.
NFS v.3 Unix security	Unix users, file mode bits	Mounting file systems, reading and writing files.

Unix User ≠ OCI User

Remember that users in UNIX aren’t the same as users in OCI - they’re not linked or associated in any way. The OCI policy layer doesn’t govern anything that happens inside the file system, the UNIX security layer does. Conversely, the UNIX security layer doesn’t govern creating file systems or mount targets in OCI.

Mount Target

Mount Target is an NFS endpoint that lives in a subnet.  It has the following features:

• Must be in the same availability domain as the file system.
• When you create a file system, you can create a mount target or use an existing one.
• Each mount target has an export set
• A collection of one or more exports that control what file systems the mount target exports using NFSv3 protocol and how those file systems are found using the NFS mount protocol. 
• The information stored in an export includes the file system OCID, export path, and export options. For more information, see Working with NFS Export Options.
• Exports and export sets are managed through the Console by the FSS. More advanced configuration options for exports and export sets are available in the Command Line Interface (CLI) and API.
• Provides the IP address or DNS name that is used in the mount command when connecting NFS clients to the FSS. 
• Requires three private IP addresses in the subnet (don't use /30 or smaller subnets for the FSS)
• Should be placed in its own subnet
• Placing NFS clients and mount target in the same subnet can result in IP conflicts, as users are not shown which private IP;s are used for mount target.
• Mount Target Limits
• By default, you can create two mount targets per account per availability domain.

File System

File System is a way that your Compute instances use to organize files on your storage.  You can create a shared file system in the cloud using the FSS.  To access your file systems, you need to create a new (or use an existing) mount target.

When you use the Cloud Console, creating a file system also creates a mount target that your Compute instances use to access and write to the file system.   Once set up, file systems can be accessed from any OCI BM/VM instance in the same or other regions.  They can also be accessed from your on-premise environment using FastConnect or IPSec VPN.^[3]

1 Mount Target : N File systems

Using the API or the Command Line Interface (CLI), you can create file systems and mount targets independently of each other.  Both file systems and mount targets are associated with an availability domain.

Once a mount target is created, multiple file systems can be associated with it. You can create up to 100 file systems per mount target.  However, no two file systems associated with the same mount target can have overlapping export path (e.g., FS paths like /example and /example/path are not allowed)

Paths in File Systems

In the discussions below, we will use this mount command example:

sudo mount 10.0.0.6:/example/path /mnt/mountpointA 

where  

10.0.0.6 is the mount target IP address and /example/path is the export path

There are three kinds of paths that are used in the FSS:

• Export Paths
• Specified when a file system is associated with a mount target
• Appended to the mount target IP address, and used to mount (logically attach) to the file system
• Used only to distinguish one file system from another within a single mount target
• Syntax Rules
• Must start with a slash (/) followed by a sequence of zero or more slash-separated elements.
• Cannot end in a slash
• Cannot be a period (.) or two periods in sequence (..)
• Cannot exceed 255 bytes
• Mount Point Paths
• Are paths within a client instance to a locally accessible directory to which the remote file system is mounted
• /mnt/mountpointA is the path to the directory on the client instance on which the external file system is mounted.
• File System Paths
• Are paths to directories within the file system, and contain the contents of the file system. 
• For example, snapshots of the file system can be accessed using the file system path, under the file system's root directory at .snapshot/name.
• The following example shows the path to a snapshot called 'January 1' when navigating from the instance:
• /mnt/mountpointA/.snapshot/January1

Creating a File System

Follow these steps to create a File System on OCI:

1. From the Core Infrastructure, select File Storage. 
2. Choose a compartment in which you have permission to work. 
3. Click Create File System. 
4. In the Create File System dialog box, provide the following information: 
• Enter a name for the File System. 
• Choose an availability domain. 
• Choose Edit Details for Mount Target
• Choose CREATE MOUNT TARGET (or you can SELECT AN EXISTING MOUNT TARGET)
• Enter a name for the Mount Target 
• Choose a VCN for Mount Target
• Choose a subnet for Mount Target (e.g., FSS-subnet as in the video)
• Click Show Advanced Options
• Before you can mount this file system, you must configure security rules to allow traffic to subnet FSS-subnet.  
• Choose IP ADDRESS
• Choose HOSTNAME
• Choose PATH
• Path name cannot be currently in use in the Mount Target
• Choose MAXIMUM FREE SPACE (IN GiB)
5. Click Create File System.

Configuration Details

• File System Information
• Name
• The file system name doesn't need to be unique; an Oracle Cloud Identifier (OCID) uniquely identifies it.
• Availability domain 
• The availability domain in which you want to run the File System.
• Mount Target Information
• File systems must be associated with a mount target in order to be mounted by any instance.
• By default, the mount target is created in your current compartment.
• The mount target must be in the same availability domain as the file system. You cannot change the availability domain.
• Virtual Cloud Network
• The first VCN listed in the selected compartment is used as default.
• Subnet
• The first subnet listed in the selected availability domain is used as default.
• FSS requires ingress TCP ports 111, 2048,2049, and 2050, in addition to standard ingress UDP ports 111 and 2048.  Opening these ports enables traffic from Solaris, Linux, and Windows instances.  Read Configuring VCN Security List Rules for File Storage for more information.
• Don't place NFS client(s) and mount target in the same subnet
• IP Address (Optional)
• You can specify an unused, local, private IP address, for example, between 10.0.0.2 and 10.0.0.254 for the new mount target.
• Hostname (Optional)
• You can specify a hostname you want to assign to the mount target.
• Which will be used to created a fully qualified domain name (FQDN) which cannot be changed in this dialog.
• Path Name (Optional)
• You can replace the default export path (default path using new file system's name) with a new one.
• Maximum Free Space (in GiB; Optional)
• Specifies the maximum free space (default: 8 exabytes) reported to applications by the FSS.
• Some applications fail to install because a capacity check reports too much available capacity (i.e. the default 8 exabytes). 
• Setting the Maximum Free Space reported as available to a value acceptable by your application prevents this issue. 
• Setting the maximum free space affects each file system associated with the mount target. 
• Setting the maximum free space does not limit the amount of data you can store.
• Export Options (Important)
• Enable the "Require Privileged Source Port" export option. For more information, see Working with NFS Export Options.

Summary

File Storage Service is an AD-local service, available in all OCI regions and Availability Domains, which supports

• NFS v.3
• Network Lock Management (NLM) for file locking
• Full POSIX semantics
• Data Protection
• Snapshot capabilities; 10,000 snapshots per file system
• Security
• 128-bit, data-at-rest encryption for all file systems & metadata
• Management Options
• Console , APIs, CLI, data-path commands, and Terraform

You can create 100 file systems and 2 mount targets per AD per account.  But you can request an increase. See Service Limits for a list of applicable limits and instructions for requesting a limit increase.

References

1. OCI Storage White Papers 
2. Best Practices for Deploying HA Architecture on OCI  
3. OCI―Knowing Virtual Cloud Network Basics (XML and More)
4. Configuring VCN Security List Rules for File Storage (OCI)
5. How EBS can be deployed on OCI
6. VCNs and Subnets (OCI)
7. Developer Tools (OCI)
8. OCI Security Best Practices
9. Learn more about VCNs
10. About Security (OCI)
11. Documentation 
12. Blogs
13. API Documentation (OCI)

EXT3-fs warning: checktime reached, running e2fsck is recommended

From Linux syslog^[1] (i.e., /var/log/messages), we have found the following message:

• EXT3-fs warning: checktime reached, running e2fsck is recommended 

And, from the file system disk space report (i.e., "df -k"), we find:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             61189192  31629588  26401228  55% /
/dev/sda1               101086     21650     74217  23% /boot
tmpfs                 16472072         0  16472072   0% /dev/shm
/dev/sdb              70440240  35203348  31658524  53% /data

So, our Linux platform has three different file systems:

• /dev/sda1
• /dev/sda3
• /dev/sdb

What is /dev/sda1^[2,5]?

For Linix device naming,

• hd
• It designates IDE devices
• sd
• It designates SCSI devices including kernel-level emulation of SCSI devices, like USB devices or, in some cases, CD-RW drives.

The a and the b's, etc .. (i.e., sda, sdb)are the equivalent to the concept of hda, and hdb, etc.  When different devices are plugged in, they are mapped to sda, sdb, etc. For detailed information about these plug-in devices, we can find them in the boot message, which can be generated by:

• $dmesg > boot.messages

From the boot message, we have found the following entries:

scsi0 : aacraid Vendor: Sun Model: ssssss-xdddd-dd Rev: V1.0 Type: Direct-Access ANSI SCSI revision: 02 SCSI device sda: 143134720 512-byte hdwr sectors (73285 MB) sda: Write Protect is off sda: Mode Sense: 06 00 10 00 SCSI device sda: drive cache: write through w/ FUA SCSI device sda: 143134720 512-byte hdwr sectors (73285 MB) sda: Write Protect is off sda: Mode Sense: 06 00 10 00 SCSI device sda: drive cache: write through w/ FUA sda: sda1 sda2 sda3 sd 0:0:0:0: Attached scsi removable disk sda Vendor: Sun Model: solaris root Rev: V1.0 Type: Direct-Access ANSI SCSI revision: 02

 .

 .

 .

EXT3 FS on sda3, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on sdb, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 8289532k swap on /dev/sda2.  Priority:-1 extents:1 across:8289532k

So, we know our devices use EXT3 FS.

What Is EXT3 FS^[3]?

EXT3 FS, or third extended filesystem, is a journaled file system that is commonly used by the Linux kernel.  It is the default file system for many popular Linux distributions.

EXT3 adds the following features to EXT2:

• A journal
• Online file system growth
• Htree indexing for larger directories

Without these features, any EXT3 file system is also a valid EXT2 file system. This situation has allowed well-tested and mature file system maintenance utilities for maintaining and repairing EXT2 file systems to also be used with EXT3 without major changes. The EXT2 and EXT3 file systems share the same standard set of utilities, e2fsprogs, which includes an fsck tool. The close relationship also makes conversion between the two file systems (both forward to EXT3 and backward to EXT2) straightforward.

With all these background information, now we are ready to run e2fsck as recommended.

Running e2fsck

In general it is not safe to run e2fsck on mounted filesystems. So, before we run e2fsck command, we have unmounted our file system.

[root@myserver bench]# umount -v /dev/sdb
/dev/sdb umounted
[root@myserver bench]# /sbin/e2fsck -v  /dev/sdb
e2fsck 1.39 (29-May-2006)
data has gone 652 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

  349958 inodes used (3.90%)
    2278 non-contiguous inodes (0.7%)
         # of inodes with ind/dind/tind blocks: 16016/1145/0
 9082105 blocks used (50.76%)
       0 bad blocks
       2 large files

  311040 regular files
   38016 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
     893 symbolic links (893 fast symbolic links)
       0 sockets
--------
  349949 files

/etc/fstab^[4]

The 6th column of fstab is a fsck option. fsck looks at the number in the 6th column to determine in which order the filesystems should be checked. If it's zero, fsck won't check the filesystem.

In our system, we find that both "/" and "/boot" filesystems are automatically checked whenever the system is rebooted:

LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2

Therefore, we don't need to fsck either /dev/sda3 or /dev/sda1.

References

1. Troubleshooting Linux with syslog
2. What is /dev/sda1?
3. ext3 (Wikipedia)
4. How to edit and understand /etc/fstab - 1.1
5. linux sda vs hda ?