RDG—Oracle Analytics Cloud Remote Data Gateway

You can connect to remote on-premise data sources from Oracle Analytics Cloud. This enables you to deploy Oracle Analytics Cloud with large on-premise data sets without migrating the data to the cloud. Users can analyze the data in data visualizations, and in reporting dashboards and analyses.

Oracle Analytics Cloud Data Gateway (which supersedes Remote Data Connector) enables you to query on-premises data sources from Oracle Analytics Cloud.

Note that a new feature named private access channel (PAC) was also introduced for the Oracle Analytics Cloud (OAC) Gen 2 instances since 5.9.  Here are the high-level descriptions of of it:^[14]

Private Access Channel compliments and provides an alternative to Remote Data Gateway (RDG) for the private data sources it supports. Both PAC and RDG may be used within the same instance of OAC. There may be sources that PAC supports and RDG does not and vice-versa.

Figure 1.  Data Gateway Architecture (Source: [11]) . The callouts are

    1. Oracle Analytics Cloud issues and queues queries using Data Gateway queue.

    2. The agent looks for queries to process via long-polling.

    3. The agent executes the queries.

    4. The agent sends the query results to Oracle Analytics Cloud.

Data Gateway Architecture

Figure 1 shows a high-level diagram of the architecture for Data Gateway. 

Oracle Analytics Cloud communicates through a firewall with an on-premise database using HTTPS, with an agent installed in the on-premise environment that channels database queries between Oracle Analytics Cloud and the on-premise database. 

On-premise Environment

You install Data Gateway in a subnet that gives visibility to both Oracle Analytics Cloud and the target data sources. Your network needs to allow outgoing (egress) traffic from the node where Data Gateway is installed to the public internet on port 443 so that Data Gateway can communicate with Oracle Analytics Cloud. In addition, the network needs to allow outgoing (egress) traffic from the Data Gateway agent to the data source. 

For example, you might test the network by opening a browser on the node where Data Gateway is installed and connect to Oracle Analytics Cloud. You might also test the connection from the same node to the data source using a generic JDBC tool.

Data Gateway regularly communicates with Oracle Analytics Cloud to see whether Oracle Analytics Cloud has queries that need processing, a process known as long-polling. Data Gateway makes a long-running TLS-encrypted HTTP request to Oracle Analytics Cloud and waits until Oracle Analytics Cloud has a query to process. If there're no queries from Oracle Analytics Cloud after two minutes, Data Gateway terminates and re-issues the request in order to avoid the request being identified and terminated as an idle or stale connection by the network.

The HTTPS communication between Data Gateway and Oracle Analytics Cloud leverages the SSL certificate of your Oracle Analytics Cloud service instance. The same certificate is used to encrypt your browser connections to Oracle Analytics Cloud.

Data Gateway vs Remote Data Connector

Data Gateway replaces the Remote Data Connector utility that was used in earlier releases. Although you can still use Remote Data Connector, Oracle recommend that you use Data Gateway. Data Gateway has many advantages over Remote Data Connector:

• You can deploy multiple Data Gateway agents to query the same database. This provides failover and a highly-available architecture.
• In Remote Data Connector, the communication is initiated by Oracle Analytics Cloud. In Data Gateway, the communication is initiated by Data Gateway. As a result:
  • You don't need to install anything in a network demilitarized zone (DMZ).
  • You don't have to open ports for ingress traffic in the on-premises firewall.
  • You don't have to register your Oracle Analytics Cloud as a 'safe' domain.

Note that you cannot use Data Gateway and Remote Data Connector at the same time with your Oracle Analytics Cloud instance.

Server Deployment vs Personal Deployment

Data Gateway supports two kinds of deployment:

• Server Deployment
• You can use Oracle Universal Installer to install on Linux
• You can have multiple Data Gateway agents on each node (physical or virtual)
• You can edit Data Gateway properties in 
• <Installed location>/jetty/obiee_rdc_agent.properties
• Personal Deployment
• You can copy the binary files onto MacOS or Windows
• You can edit Data Gateway properties in 
• <Installed location>\obiee_rdc_agent.properties

High Availability

High Availability is supported for Data Gateway and Oracle recommends at least two Data Gateways per Oracle Analytics Cloud instance.  

How HA is supported?

• On the Oracle Analytics Cloud side, high availability is provided natively.
• On the Data Gateway side, you set up high availability by deploying two Data Gateways for each Oracle Analytics Cloud instance.
• However, all of these agents must be capable of servicing all remote queries (that is, you cannot configure one agent to service queries for one data source only, and another agent to service queries for a different data source).

Troubleshoot Data Gateway

Here's how to troubleshoot common setup issues with Data Gateway.

Issue reported	Do this
All issues	Check that you are on the correct version of Data Gateway that matches your version of Oracle Analytics Cloud. To check the version of the Data Gateway agent, on a server deployment, refer to the version in the file <installdirectory>/inventory/registry.xml.
Agent state change failed with error: Agent name or Oracle Analytics Cloud URL not specified or the Key pair not generated	Click Save, then Enable. If the problem persists, restart the application. If necessary, check your network.
Invalid Oracle Analytics Cloud URL (Data Gateway can't communicate with Oracle Analytics Cloud)	Check that you've enabled and configured Data Gateway in Oracle Analytics Cloud Console. Make sure you can reach the Oracle Analytics Cloud URL from the environment where Data Gateway is running. For example, on Linux you might use a traceroute command, such as sudo traceroute -T -p 443 <Fully qualified domain name of your Oracle Analytics Cloud instance>. Make sure there's nothing else blocking communication through the firewall. If you're using a proxy: Stop the Data Gateway agent. On the machine where you've installed Data Gateway, configure the Data Gateway properties: In a server deployment, edit this file: <Installed location>/jetty/obiee_rdc_agent.properties In a personal deployment, edit this file: <Installed location>\obiee_rdc_agent.properties Update the following properties with the details of your internet proxy host: proxyUserName proxyPassword proxyHost (for example, www-proxy.us.oracle.com) proxyPort (for example, 80) Start the Data Gateway agent.
Test fails in Console page Remote Data Connectivity	If the test fails, this means that the Data Gateway agent can't authenticate. Possible reasons include: The agent key hasn't been copied to the Remote Data Connectivity page in Oracle Analytics Cloud Console. The agent key has been regenerated in the agent, but the new key hasn't been copied to the Remote Data Connectivity page in Oracle Analytics Cloud Console. There's no suitable network route from the agent to Oracle Analytics Cloud.

Frequently Asked Questions about Data Gateway

Please check RDG Documentation on Data Gateway Frequently Asked Questions :

• What operating systems does Data Gateway support?
• What is the Data Gateway Architecture?
• What is the difference between Remote Data Connector and Data Gateway?
• Where do I install Data gateway?
• Can I deploy multiple Data Gateway agents?
• How do I configure High Availability for Data Gateway?
• Why is Data Gateway traffic egress only?
• How does Data Gateway manage SSL certificates?
• How do I size Data Gateway?
• Where is Data Gateway running? Do I install it on a virtual machine (VM)?
• Can I use Data Gateway and Remote Data Connector with my Oracle Analytics Cloud instance at the same time?
• How is Data Gateway network traffic secured?
• Can Data Gateway limit queries that affect performance or security?
• Data Gateway doesn't limit the query row size. The query row size limit is determined by the number of Oracle Compute Units (OCPUs) that your Oracle Analytics Cloud service has.
• What is the timeout setting for Data Gateway?
• Data Gateway uses the same query timeout as Oracle Analytics Cloud. See Limits Querying Data (Data Visualization Projects, Classic Analyses and Dashboards).

Deployment of Remote Data Gateway

Read the below articles for the step-by-step instructions on the installation and deployment of remote data gateway (RDG) for Oracle Analytics Cloud:

Installing Oracle Analytics Cloud Remote Data Gateway
Preparing Linux for Oracle Remote Data Gateway
Deploying Oracle Analytics Cloud Remote Data Gateway in a Private Subnet
Deploying Oracle Analytics Cloud Remote Data Gateway in a Public Subnet
Deploying Oracle Analytics Cloud Remote Data Gateway On Premise
Deploying Oracle Analytics Cloud Remote Data Gateway using Local Peering Gateways

Deploying Oracle Analytics Cloud Remote Data Gateway in a Remote Private Subnet 

Completing the Deployment of Remote Data Gateway for Oracle Analytics Cloud

If you have subscribed to My Oracle Support, you can also check the below articles for more information:

• NOTE:2590606.1 - OAC: How To Increase Logging For Remote Data Gateway Agent in OAC
• NOTE:2574387.1 - Guidance For Using Remote Data Gateway (RDG) on Environments Upgraded From Oracle Analytics Cloud 105.2 and Lower
• NOTE:2631494.1 - OAC: Test From Remote Data Gateway(RDG) Agent Config page fails with "Invalid OAC URL" Error
• NOTES:2653783.1  - OAC RDG 5.5 : RDG Agent Configuration With Proxy Authentication Errors Out While Connecting To OAC
• NOTES:2815638.1 - All Dashboard Reports Using RPD Failed With Error: Invalid Oracle URL specified

References

1. Troubleshooting Data Gateway
2. Installing Oracle Analytics Cloud Remote Data Gateway
3. Completing the Deployment of Remote Data Gateway for Oracle Analytics Cloud
4. OAC: Test From Remote Data Gateway(RDG) Agent Config page fails with "Invalid OAC URL" Error (Doc ID 2631494.1)
5. OAC RDG 5.5 : RDG Agent Configuration With Proxy Authentication Errors Out While Connecting To OAC (Doc ID 2653783.1)
6. Configuring Oracle Analytics Cloud
7. Configure and Register Data Gateway for Data Visualization
8. Give Data Sources Access to Analytics Cloud Instances
9. Cache Management in Oracle Analytics Cloud 5.4 Onwards
10. Best Practices from Oracle Development's A‑Team
11. Connect to On-premise Data Sources
12. Connecting to Private Data Sources using Oracle Analytics Cloud Private Access Channel
• Private Access Channel compliments and provides an alternative to Remote Data Gateway (RDG) for the private data sources it supports. Both PAC and RDG may be used within the same instance of OAC.
13. Preparing Network Gateways for Private Oracle Analytics Cloud Data Sources
14. Connect to Private Data Sources Through a Private Access Channel
15. Oracle Analytics Managing Security for Oracle Analytics Server, 5.9.0 (pdf)
16. Troubleshooting Private Access Channel Connectivity for Oracle Analytics Cloud
17. Oracle Analytics Cloud supported Data Sources
18. Connecting to Oracle Analytics Cloud Private Endpoint with VPN / FastConnect
19. Manage Service Access and Security
20. Prerequisites for a Private Access Channel (PAC)

OCI―Knowing Database Service Basics

Video 1.  OCI Level 100 - Database (Demo starts at 30:20; YouTube link)

The Oracle Cloud Infrastructure (OCI) Database service (DS) offers autonomous and user-managed Oracle Database solutions:

• Fully-managed 
• Autonomous databases are preconfigured, fully-managed environments that are suitable for either transaction processing or for data warehouse workloads. 
• See OCI―Knowing Autonomous Database Basics for more information
• User-managed
• User-managed solutions are bare metal (BM), virtual machine (VM), and Exadata DB systems that you can customize with the resources and settings that meet your needs.

You can quickly provision a user-managed DB system or autonomous database. You have full access to the features and operations available with the database, but Oracle owns and manages the infrastructure:

In this article, we will cover mainly on Oracle provided user-managed solutions and use DS to mean either DB System or Database Service.

	Virtual Machine (VM)	Bare Metal (BM)	Exadata
Scaling	Storage (number of CPU cores on VM DB cannot be changed)	CPU (amount of available storage cannot be changed)	CPU can be scaled within a ¼, ½ and Full rack. Storage cannot be scaled
Multiple Homes/Databases	No, single DB and Home only	Yes (one edition, but different versions possible)	Yes
Storage	Block Storage	Local NVMe disks	Local spinning disks and NVMe flash cards.
Real Application Clusters (RAC)	Available (2-node)	Not Available	Available
Data Guard	Not Available	Available*	Available*

*: You can manually configure Data Guard on Exadata DB systems using native Oracle Database Utilities and commands.  dbcli is not available on Exadata Data systems.

Figure 1.  Exadata Database System with the Maximum Availability Architecture:

DS Features

OCI Database Service has the following features:

• Comprehensive offerings to cover all enterprise database needs
• 3 Shape Types:
• Exadata, Bare Metal, Virtual Machine 
• Each type with different shapes 
• Supports various Oracle database software editions, license types, database versions, and database workloads (OLTP vs DSS)
• Lifecycle Automation
• Provisioning, Patching, Backup & Restore
• High Availability 
• RAC & Data Guard
• Scalability
• Dynamic CPU and Storage scaling
• Security
• Infrastructure (IAM, Security Lists, Audit logs)
• Database (TDE, RMAN backup/Block volume encryption)
• OCI Platform integration
• Tagging, Limits and Usage Integration
• License Type
• License Included with OCI provisioning
• Bring Your Own License (BYOL)

With Exadata DS, all types of failures are protected against, including disk, server, network and data.

Managing DS

You can use the console to perform the following tasks:

• Launch DS 
• Which also create a database system
• Start, stop or reboot DS
• Billing continues in stop state for BM DB Systems because its attached local NVMe, but not for VM 
• Scale CPU cores (except for VM)
• Scale up storage (VM only)
• Terminate
• When you terminate a DB system or a database, all of its resources are deleted, along with any automatic backups. 
• Full backups remain in Object Storage as standalone backups. 
• You can use a standalone backup to create a new database.



Figure 2.  Db System patch (upper) vs Database patch (lower)

Patching DS

Because patching a system requires a reboot, plan to run the operations at a time when they will have minimal impact on users. To avoid system interruption, consider implementing a high availability strategy such as Oracle Data Guard.

Using the Console, API, or the database CLI (dbcli), you can perform below tasks:

• Managed/On-Demand Patching
• Automated applicable patch discovery and pre-flight checks/tests
• On Demand Patching
• N-1 patching (previous patch is available if it hasn't been applied), pre-check and patching at the click of a button
• Availability during Patching 
• For Exadata and RAC shapes, patches are rolling.
• For single node systems, Active Data Guard can be leveraged by the patch service if it is configured
• 2-Step Process
• Patching is a 2 step process.  To perform a patch operation on a
1. DB System
2. Database
• Identity and Access Controls
• Controls who can list patches, apply them, etc

Backup / Restore

If you plan to back up your DB system to Object Storage, you can use a service gateway with a private subnet or an internet gateway with a public subnet.  To recover a database from Object Storage, you can use the Console, API, or by using RMAN.^[2]

For backup and restore, OCI Database Service supports:

• Managed Backup 
• Exadata backup process requires creating a backup config file.  See Backing Up an Exadata Database for more information.
• For  BM/VM backups, see "Backup Destinations" below.
• Private Networking
• The resources in the DS can be in a private subnet and use only private IP addresses.
• Even with an internet gateway, network traffic between the DS and Object Storage does not leave the cloud and never reaches the public internet.
• Backup Destinations
• Object Storage (recommended) or Local Storage
• For Exadata backups, read Backing Up an Exadata Database for information
• For BM / VM backups, read:
• Backing Up to Oracle Cloud Infrastructure Object Storage
• Backing Up to Local Storage Using the Database CLI (only applicable to BM databases)
• Backup Options
• Automatic incremental―Runs once a day; repeats the cycle every week; retained for 30 days after you terminate the DS^[11]
• Full backups―On-demand, standalone^[10]
• Restore Options
• To the latest
• To the timestamp
• To System Change Number (SCN)

Manged Backups vs. Unmanaged Backups

• Managed backups
• See Backing Up to Oracle Cloud Infrastructure Object 
• Unmanaged backups
• The list of backups you see in the Console does not include any unmanaged backups (backups created directly by using RMAN or dbcli).

Automatic Backups (Incremental)

• Default Policy 
• Not changeable at this time
• Are written to Oracle owned object storage (not visible to customers)
• Backup window is defined by Oracle
• New backup window time will be between midnight and 6 AM in the time zone of the DS's region
• To migrate yours to the new behavior, contact Oracle Support 
• All backups are encrypted with the same master key used for Transparent Data Encryption (TDE) wallet encryption.  
• Automatically retried
• Oracle will get notified if the job is stuck

High Availability (HA)

• Cloud Infrastructure HA
• Region with 3 Availability Domain architecture
• While the availability domains provide the facility for high availability, regions provide the basis for disaster recovery. 
• Fully redundant and non-blocking Networking Fabric
• Database HA
• Disk Redundancy
• Normal (2-way mirroring) vs High (3-way mirroring―recommended for production) 
• Database RAC option
• Automated Data Guard within and across ADs

Read Oracle's whitepaper on HA for more information.

Scalability

• Exadata 
• Two kinds of scaling operations are supported for an Exadata DB system:
• Scaling within an Exadata DS lets you modify compute node processing power within the system.
• Scaling across Exadata DS configurations lets you move to a different configuration, for example, from a  ¼  to a ½ rack.
• To learn how to scale an Exadata DS, see To scale an Exadata DB system.
• Bare Metal 
• # of CPU cores can be scaled
• Total raw storage is fixed based on a chosen shape
• However, the amount of usable storage on the BM DS can be affected by options like 2- or 3-way mirroring and the space allocated for data files.
• Virtual Machine 
• 1-node or 2-node RAC 
• Storage can be scaled
• For 2-node RAC VM DB systems, storage capacity is shared between the nodes.
• # of CPU cores on an existing VM DS cannot be changed.

References

1. Overview of Database Service (OCI)
2. Recovering a Database from Object Storage
3. Developer Tools (OCI)
4. Updating a DB System (OCI)
• Includes information and instructions on how to update the OS of a bare metal or virtual machine DB system.
5. Updating an Exadata DB System (OCI)
6. Securing Database (OCI)
• Includes information on security best practices for databases in Oracle Cloud Infrastructure.
7. OCI Database Service Troubleshooting 
• Backup Failures
• Patching Failures
8. Oracle Cloud Infrastructure Object Storage FAQ
9. Oracle Database CLI Reference (dbcli)
10. Standalone Backups (OCI)
11. Backing Up to Oracle Cloud Infrastructure Object  (OCI)
12. Migrating Databases to the Cloud (OCI)
13. Developer Tools (OCI)
14. Introduction to Oracle Automatic Storage Management (ASM)
15. Best Practices for Deploying High Availability Architecture on Oracle Cloud Infrastructure (must read)
16. OCI API Documentation 

Apache Hadoop HDFS一Knowing the Basics

Hadoop HDFS (Hadoop Distributed File System) is a distributed Java-based file system for storing large volumes of data. It is designed:

• To be a scalable, fault-tolerant, distributed storage system 
• To be the data management layer of Apache Hadoop
• Hadoop (data management layer) = HDFS + YARN
• YARN provides the resource management 
• HDFS provides the distributed storage for big data
• HDFS works closely with a wide variety of concurrent data access applications, coordinated by YARN.
• To span large clusters of commodity servers
• HDFS will “just work” under a variety of physical and systemic circumstances.
• HDFS cluster = NameNode + DataNodes

In this article, we will use Apache Hadoop HDFS from the Hortonworks Data Platform (HDP: version 2.4.2) in the discussion.  For HDFS High Availability (HA) feature, our reference is based on [2].

HDFS Cluster

An HDFS cluster is comprised of a NameNode, which manages the cluster metadata, and DataNodes that store the data.  Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable .

You can follow the instructions here to format and start HDFS on Hortonworks Data Platform. HDFS can be accessed from applications in many different ways. Natively, HDFS provides a FileSystem Java API for applications to use. A C language wrapper for this Java API is also available. In addition, an HTTP browser can also be used to browse the files of an HDFS instance. Work is in progress to expose HDFS through the WebDAV protocol.  For more information, read here.^[3,10,11]

Name Node

High-level summary of Name Node which it:

• Provides high availability (HA) using redundant Name Nodes^[2]
• NameNode (active)
• Secondary NameNode (standby)
• Maintains the following two metadata files (or checkpoint files):
• fsimage file
• Holds the entire file system namespace,^[12] including the mapping of blocks to files and file system properties
• editlog file
• Holds every change that occurs to the filesystem metadata

Namenode Web UI 

To smoke test your NameNode server, you can use the following URL^[7,11]

http://$namenode.full.hostname:50070

to determine if you can reach the NameNode server with the browser. If successful, you can also select the Utilities menu to "browse the file system".

High Availability

The HDFS High Availability feature (vs. another new HDFS Federation feature) addresses the SPOF problem by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance.

If your individual IDs of NameNodes are nn1 and nn2, you can get their service status using the following command:^[3]

$ sudo -u hdfs hdfs haadmin -getServiceState nn1
active
$ sudo -u hdfs hdfs haadmin -getServiceState nn2
 standby

Metadata Files

When NameNode starts up, it reads FsImage and EditLog files from disk, merges all the transactions present in the EditLog to the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage.

Metadata files are stored at: 

• ${dfs.namenode.name.dir}/edits
• ${dfs.namenode.name.dir}/fsimage

where dfs.namenode.name.dir property can be configured in hdfs-site.xml.^[8]

Data Node

High-level summary of Data Node:^[4]

• Scalable Storage
• HDFS cluster storage scales horizontally with the addition of DataNodes
• Minimal data motion
• Hadoop moves compute processes to the data on HDFS and not the other way around. 
• Processing tasks can occur on the physical node where the data resides, which significantly reduces network I/O and provides very high aggregate bandwidth.
• Data Disk Failure一Heartbeats and replication
• Each DataNode sends a Heartbeat message to the NameNode periodically. 
• If NameNode detects a DataNode stop sending Heartbeat message, it marks DataNode as dead and stop forwarding new IO requests to them.
• The NameNode constantly tracks which blocks need to be replicated and initiates replication whenever necessary. The necessity for re-replication may arise due to many reasons: 
• a DataNode may become unavailable
• a replica may become corrupted
• a hard disk on a DataNode may fail
• the replication factor of a file may be increased
• Data Rebalancing
• HDFS automatically move data from one DataNode to another if the free space on a DataNode falls below a certain threshold
• Data Integrity一checksum
• When a client creates an HDFS file, it computes a checksum of each block of the file and stores these checksums in a separate hidden file in the same HDFS namespace. 

References

1. Hadoop Distributed File System (HDFS)
2. HDFS High Availability Using the Quorum Journal Manager
• You can enable High Availability using multiple NameNodes either with 
• a shared storage on NFS or 
• a distributed edit log (called Journal).
3. HDFS Commands Guide (Apache Hadoop) 
• All HDFS commands are invoked by the bin/hdfs script and can be grouped into:
• User commands
• Administrator commands
• Debug commands
4. HDFS Architecture (Apache Hadoop) 
5. Apache Hadoop
6. HDFS Federation (Hortonworks)
• In order to scale the name service horizontally, federation uses multiple independent Namenodes/namespaces. 
7. HDFS Ports (Hortonworks)
8. Apache Ambari一Knowing the Basics (Xml and More)
9. hdfs-default.xml (2.7.1)
10. FileSystem Shell - Apache™ Hadoop
11. Hadoop NameNode Web Interface
12. Namespace (HDFS)
• Consists of directories, files and blocks.
• It supports all the namespace related file system operations such as create, delete, modify and list files and directories.
13. Hadoop DistCp Guide
• Copy file or directories recursively
14. All Cloud-related articles on Xml and More