In this Document
APPLIES TO:Oracle E-Business Suite Technology Stack - Version 11.5.10.2 and laterInformation in this document applies to any platform. ABSTRACTThis troubleshooting guide is provided to assist in the configuration and troubleshooting of the URL firewall used in Oracle E-Business Suite 11i and Release 12 DMZ topologies. The recommended prerequisite reading for this note is Note:364439.1 and Appendix E: "Configuring the URL Firewall" of Notes:287176.1 and 380490.1.Note:287176.1-DMZ Configuration with Oracle E-Business Suite 11iThe configuration and troubleshooting of the URL firewall can be very complex, but the vast majority of URL firewall problems are easily resolved with a general understanding of how the URL firewall works as explained in this note. The details of this note are geared primarily towards 11i, but the majority of the concepts apply to Release 12 as well. HISTORYCreate Date 27-SEP-2007 Update Date 26-JAN-2018 DETAILSGeneral OverviewAs described in Note:287176.1-"Oracle E-Business Suite 11i Configuration in a DMZ" (and R12's Note:380490.1), the purpose of the URL Firewall is to ensure that only URLs required for the externally exposed functionality can be accessed from the Internet. The URL firewall is implemented as a whitelist of URLs required in the form of rewrite rules interpreted by the apache web server's (Oracle HTTP Server powered by Apache) mod_rewrite module. Any URL request that is not matched in this whitelist is refused and produces the familiar "Gone" error as illustrated below.GoneThe only mechanism, thus far, in Release 11i and Release 12 that produces this GONE message is the URL firewall. Disabling the URL FirewallThe collection of rewrite rules that make up the URL firewall are all combined into a single configuration file, url_fw.conf, and this file is enabled by the following line at the bottom of the main, 11i, Apache configuration file httpd.conf. For example:#Configuration file containing Aceess Restrictions in a DMZ configuration.By default, only the tiers declared as EXTERNAL by the profile option "Node Trust Level" will have an active URL firewall. The URL firewall is automatically disabled on INTERNAL tiers by AutoConfig. AutoConfig will comment out the above reference to the file "url_fw.conf" on INTERNAL tiers, but on EXTERNAL tiers the line will be uncommented and therefore active as seen in the above example. Certainly, the quick and ugly fix to getting past any of the "Gone" errors is simply to comment this line out manually and bounce apache, since commenting out the reference to the file url_fw.conf within httpd.conf effectively disables the URL firewall entirely. This is often a reasonable, short-term fix for troubleshooting, but certainly a properly configured URL firewall is preferred for security reasons. While Release 12 has an AutoConfig variable for permanently commenting out the reference to url_fw.conf, Release 11i at the time of this writing does not. Therefore, AutoConfig will re-enable the URL firewall on every EXTERNAL tier when it is run unless a customization is in place to prevent it. URL Firewall VersionsThe collection of rewrite rules that make up the URL firewall in the file url_fw.conf is created based on the url_fw.conf template that comes with every AutoConfig rollup patch. Throughout the revision history of the url_fw.conf, various bugs have been logged to fix rewrite rules that were malformed, missing, or newly required by product changes. Therefore, you can save yourself a lot of rediscovery work by getting on the latest AutoConfig patch version so that the latest URL firewall rewrite rules are already in place. Even if you can't apply the latest AutoConfig patch, you should at least download it to review the url_fw.conf that comes with it to see if a fix for your particular GONE problem is already there.At the time of this writing, the latest AutoConfig rollup patch is: 9535311-TXK AUTOCONFIG AND TEMPLATES ROLLUP PATCH USee the following note for more information on AutoConfig: Note:165195.1-Using AutoConfig to Manage System Configurations with Oracle Applications 11i Analysis MethodsThe fundamental approach to troubleshooting "Gone" errors is to get a copy of the access_log for the apache server on the external tier. The "Gone" errors have the HTTP status code of 410 and indicate the URL that was refused. Once the rejected URL is known, we can start to determine why it was rejected.The problem is generally either that an existing whitelist pattern has not been uncommented (activated) or that there is no pattern yet for the required URL. The latter is quite common as customers often do not have the latest version of the url_fw.conf file and even if they do, sometimes there is a lag between when a URL is introduced by a product group and when it appears in the next AutoConfig patch with url_fw.conf. Analysis ExamplesHere are some classic examples and their solutions, grouped by category:1. The URL is legitimately blocked because the action is not allowed.In many cases, a "Gone" message is received because the user attempted to access a URL that has been disallowed intentionally. Getting the GONE message in this instance is not an error and is intended behavior.For example, when troubleshooting apache/mod_jserv using a classic note such as Note:230688.1, you run the "Hello" test and see the following instead of the intended success message: Here is the matching line in the $IAS_ORACLE_HOME/Apache/Apache/logs/access_log: 10.1.2.3 - - [26/Sep/2007:16:15:12 -0600] "GET /Hello HTTP/1.1" 410 400 In this case, there is no RewriteRule within the url_fw.conf file for Internet users wishing to run the "Hello" test. This is to be expected since the external tier is meant to be locked down to allow only specific, selected actions. See the "Advanced Analysis and Configuration" section of this note for correcting these types of problems where you have some custom functionality you want to be made allowed. 2. The URL is legitimately blocked because it contains a typographical error.The correct URL for accessing the iRecruitment home page is (as an example):http://bigserver.us.oracle.com:8002/OA_HTML/IrcVisitor.jspFollowing is part of the normal access_log response for the success that is automatically followed by a number of other GETS not shown here that eventually build the iRecruitment welcome page. Note that the status code is no longer 410-Gone: 10.1.2.3 - - [26/Sep/2007:17:09:28 -0600] "GET /OA_HTML/IrcVisitor.jsp HTTP/1.1" 302 443Recognizing an improperly formed URL takes some experience, but here are some examples: Bad URL Example 1: Here's what happens when the "/OA_HTML/" is omitted: Test URL: http://bigserver.us.oracle.com:8002/IrcVisitor.jsp access_log Response: 10.1.2.3 - - [26/Sep/2007:17:04:56 -0600] "GET /IrcVisitor.jsp HTTP/1.1" 410 400 Bad URL Example 2: Here's another example of a typo'd URL and the rightful "Gone" response: Test URL: http://bigserver.us.oracle.com:8002//OA_HTML/IrcVisitor.jsp access_log Response: 10.1.2.3 - - [26/Sep/2007:17:10:19 -0600] "GET //OA_HTML/IrcVisitor.jsp HTTP/1.1" 410 400 To be Internet friendly, many customers set the starting page URL to redirect to the Internet facing product they want Internet users to use. For example, after configuring the initial page for iRecruitment, the iRecruitment welcome page will come up with the user typing simply "http://bigserver.us.oracle.com" and nothing else. See Note:287176.1, Appendix E, for more information on configuring the initial page. 3. The URL is legitimately blocked because an underlying profile option contains a typo.This is very similar to the problem of a user typing a URL incorrectly, but in these cases, the URL is generated by Oracle code acting on a profile option value that was entered incorrectly by a System Administrator. Identifying these takes experience too, but the following examples illustrate the concept.Bad URL Profile Option Example 1: Applications Web Agent missing the trailing "/pls/": This illustrates the URL firewall masking an underlying problem. In this case, the user was attempting to access online help from iSupplier and the help window was blocked by the GONE error message. Certainly attempts could be made to allow the URL, but a key concept in URL firewall troubleshooting is seen here: Make sure the problem is actually with the URL firewall! With the URL firewall disabled, the attempt to get online help fails instead with a "404-Not Found" that changes to the URL firewall clearly won't help. When looking at the access_log for results of a navigation, use the timestamp and client IP address to guide you. With experience, you will recognize that the GET here returns a 404 (page not found) because the CORRECT URL to call is "/pls/dad/fndgfm...". 10.1.2.3 - - [27/Aug/2007:14:31:02 -0400] "GET /fndgfm/fnd_help.get/US/POS/@isp_index HTTP/1.1" 404 243The URL firewall rightfully blocks this, but without the URL firewall it fails with a "404-Page Not Found" because this URL is missing the "/pls/dad" in front of the /fndgfm/ . Online help is controlled primarily by the profile option "Applications Web Agent". A good query for listing out the values of profile options everywhere they are set is available in Note:364439.1 and will help determine where (user level, server level, servresp level, etc.) the profile option is set incorrectly. Bad URL Profile Option Example 2: Applications Servlet Agent missing the trailing "/oa_servlets/": When attempting to logout, the GONE error appeared. What we see in from the access_log is that the new SSO style logout starts with a GET for OALogout and that redirects to oracle.apps.fnd.sso.AppsLogout. With the URL firewall in place, all we get is the GONE error. A cursory examination of the url_fw.conf does show that the following RewriteRules are correctly uncommented and therefore this SHOULD work: RewriteRule ^/OA_HTML/OALogout\.jsp$ - [L]But here is the actual access_log - note the subtle difference in that second GET : In this case, we see a GET and we see the corresponding RewriteRule. They MUST match or else the GONE error will appear. Notice that the RewriteRule tells us that it is looking for the pattern to start with "/oa_servlets/oracle", but the GET has just "/oracle/...". With experience, one can tell that the profile option in use here is the Applications Servlet Agent and in the case here, it was missing the trailing "oa_servlets" at the user level for several obscure, but very vocal, users. Again, the profiles query from Note:364439.1 was used to identify the incorrectly set profile options. 4. A good URL is blocked because the matching RewriteRule is still commented out.This is the most fundamental of the URL firewall problems. Appendix E (Configuring the URL Firewall) of Note:287176.1 for 11i and Note:380490.1 for R12 covers this broadly, but we can illustrate this below.Suppose the objective was to implement iRecruitment on the external server. The URL to access the iRecruitment home page, as in the first example, is: http://bigserver.us.oracle.com:8002/OA_HTML/IrcVisitor.jspIf you get the GONE error for this or other related iRecruitment navigations, you almost certainly failed to activate the set of RewriteRules for iRecruitment. Within url_fw.conf, there are several sections for each of the externally certified products. You must MANUALLY uncomment each section of each product you want to use. For example, for iRecruitment the relevant section is: #================================================================At this point in this paper, it should be easily recognized that the first RewriteRule (above) allows the IrcVisitor.jsp URL call to be processed further. If the RewriteRule was commented out (having a '#' symbol as the first non-space character in the line), then the browser's call to IrcVisitor would return the GONE error. 5. A good URL is blocked because the RewriteRule does not exist.This category is last in the list because it is the least common and the most difficult to fix. After determining that the problem is NOT caused by the above four categories, you may find that certain functionality still leads to the "GONE" message.The only mechanism, thus far, in 11i that produces the GONE message is the URL firewall, and if you are getting GONE messages for legitimate navigations even after properly configuring the latest version of url_fw.conf, this is likely a bug and a TAR/Service Request should be logged with Oracle Support for further evaluation. The ideal TAR for a URL firewall issue will include a written description of the navigation within applications that leads to the GONE message, screenshots that go with the written description, and, most important, the access_log showing the HTTP 410 return code. If the URL firewall is preventing legitimate functionality, Oracle support can log a bug with development to get new RewriteRules added to the url_fw.conf template. As a temporary fix, you can attempt to write your own RewriteRule to get by. See the Advanced Analysis and Configuration, below. Advanced Analysis and ConfigurationAs stated in Note:287176.1, Appendix E:There could be scenarios where you may want to add custom or change the currentAt least 95% of the URL firewall problems are resolved using the first four methods listed above. The need to delve into a hardcore analysis and manual writing of RewriteRules such as is documented below is RARE, but added for completeness. URL Rewriting with mod_rewrite depends a great deal upon a solid knowledge and understanding of regular expressions as the "RewriteRule" uses regular expressions in patterns extensively. A course in mod_rewrite and regular expression syntax is beyond the scope of this guide, but the author found the following URL's to be very helpful in providing background information: http://httpd.apache.org/docs/1.3/mod/mod_rewrite.htmlWith a basic understanding of regular expressions, it is fairly easy to make new rules using the existing rules in the url_fw.conf as examples. Example of making a custom RewriteRule As referenced above in analysis example 1, the "Hello" test is blocked by the URL firewall. To allow the Hello test, a custom RewriteRule can be written and added to url_fw.conf. Using the existing RewriteRules as a guide and the above URLs for a basic understanding, the resulting rule for allowing Hello is as follows: RewriteRule ^/oa_servlets/Hello$ - [L] This rule can then be inserted (ideally by following the AutoConfig customization as explained in Metalink Note 270519.1) into the url_fw.conf file and after the next apache restart the Hello test is allowed: See the section below for more examples of custom RewriteRules. Tracing mod_rewriteThe author personally has yet to find this necessary, but mod_rewrite does have its own logging and this is supported in the Oracle version of the Apache configuration files. In httpd.conf, find the following lines:RewriteLog /space2/v115cu2/viscora/iAS/Apache/Apache/logs/rewrite.logNote that there are a set of these lines, commented out, towards the top of the file. The ACTIVE set of these are towards the bottom of httpd.conf and are the ones that count. Even if you uncomment the first set of these lines, the values set by the last set of lines are the ones that count. The RewriteLogLevel takes a number from 0 to 9 where zero is not at all and nine is verbose. The commonly observed, zero-byte, rewrite.log in the Apache/logs directory is the result of a RewriteLogLevel setting of zero. The mod_rewrite log is useful in determining what string was received by mod_rewrite and to what pattern it is matching as it goes through the rewrite list one by one. Here's an abbreviated excerpt from the Hello test: 10.3.4.5 - - [27/Sep/2007:14:07:51 -0600]- Commonly Requested URL RewriteRules (examples from the wish list)In this new section, we cover some favorite examples of custom RewriteRules to add functionality to the DMZ tier which was not authorized by the provided url_fw.conf. In each case, I list the access_log entry that is used for identifying the blocked URL and a suggested RewriteRule to allow the URL.1. Cannot review reports (web report review agent) via the DMZ tier.By default, this was never allowed because such reports generally contain sensitive data that should only be accessed via an internal tier. Making use of this rule assumes that the underlying FNDFS utility can make a sqlnet connection from the external tier to the database tier and that the customer has alternative security in place beyond the DMZ tier to keep such information from prying eyes. The last part of that rule is always the case when customizing the URL firewall to allow things that may have been disallowed deliberately. With that caveat, the following example is given:10.100.2.5 - - [28/Jul/2010:16:07:28 -0700] "GET /OA_CGI/FNDWRR.exe?temp_id=1155535699 HTTP/1.1" 410 400
RewriteRule ^/OA_CGI/FNDWRR\.exe$ - [L]
2. In Release 12.x, the OPMN pings to the web tier get a status of 410.In Release 12, processes such as the Apache (HTTP_Server) are managed by the OPMN. The OPMN then sends pings at the default interval of 20 seconds to confirm that the HTTP_Server is up. Since the url_fw.conf does not already have a RewriteRule to accommodate this, a status of 410 is received. This causes no problems though because the "410-Gone" response is still taken to mean that the HTTP_Server is alive and responding.
127.0.0.1 - - [20/Jul/2011:22:56:07 +0000] "HEAD /index.html HTTP/1.1" 410 0 0 "-" "-"
127.0.0.1 - - [20/Jul/2011:22:56:27 +0000] "HEAD /index.html HTTP/1.1" 410 0 0 "-" "-" 127.0.0.1 - - [20/Jul/2011:22:56:47 +0000] "HEAD /index.html HTTP/1.1" 410 0 0 "-" "-" If you prefer a status of "200-OK", the custom rule is simply the following:
RewriteRule ^/index.html$ - [L]
3. Correcting typos such as "oracsle" and "/OA-sHTML/".This example was complex enough to merit a separate note and just a mention here. In short, valid links were somehow distorted to result in URLs that were then rejected by the URL firewall. These were accommodated, as a workaround, using RewriteRules and were much later determined to be a valid bug and fixed. For details, see Note:1109537.1-Logout Link In E-Business Suite Gets '404-PageNotFound' or '410-Gone' with oracsle and /OA_sHTML/ (typo URL)4. The 11i SSHR (Self-Service Human Resources) RewriteRules Seem to be Missing
#================================================================
#Include URLs for SSHR (Self Service Human Resources) #================================================================ RewriteRule ^/OA_HTML/BneApplicationService$ - [L] RewriteRule ^/OA_HTML/BneViewerXMLService$ - [L] RewriteRule ^/OA_HTML/BneDownloadService$ - [L] RewriteRule ^/OA_HTML/BneUploaderService$ - [L] RewriteRule ^/OA_HTML/BneTemplateService$ - [L] RewriteRule ^/OA_HTML/BneTemplateRedirectService$ - [L] RewriteRule ^/OA_HTML/paydisplayPDF\.jsp$ - [L] 5. The 11i iRecruitment RewriteRules, Such as for Accepting Offer Letters, are Missing
#================================================================
#Include URLs for product IRC (iRecruitment) # IRC Product Pages # jsp - external only #================================================================ RewriteRule ^/OA_HTML/IrcVisitor\.jsp$ - [L] # mod_plsql --- only IRC.C # RewriteRule ^/pls/[^/]*/irc_web.show_vacancy$ - [L] RewriteRule ^/OA_HTML/IrcVisitor\.jsp$ - [L] RewriteRule ^/OA_HTML/IrcEmpVisitor\.jsp$ - [L] RewriteRule ^/OA_HTML/xdo_doc_display\.jsp$ - [L] RewriteRule ^/OA_HTML/JobPositionSeeker\.xsl$ - [L] RewriteRule ^/OA_HTML/IRCRESUMEUK1\.xsl$ - [L] RewriteRule ^/OA_HTML/IRCRESUMEUK2\.xsl$ - [L] RewriteRule ^/OA_HTML/IRCRESUMEUS1\.xsl$ - [L] RewriteRule ^/OA_HTML/IRCRESUMEUS2\.xsl$ - [L] RewriteRule ^/OA_HTML/IRCRESUMEUS3\.xsl$ - [L] 6. The R12.1.x Seeded Oracle Help RewriteRules are Missing
#================================================================
# Include Help - RewriteRule ^/OA_HTML/help$ - [L] RewriteRule ^/OA_HTML/help/topics/iHelp/HelpServlet/.*$ - [L] RewriteRule ^/OA_HTML/help/state$ - [L] RewriteRule ^/OA_HTML/help/state/.*$ - [L] #================================================================ SUMMARYWhen deploying products in the DMZ, it is not unusual to see the "GONE" error from time to time. The GONE text comes from the URL firewall, is specified in the file url_fw.conf, and generally indicates that the URL that was invoked is NOT a URL that has been allowed by the URL firewall's whitelist. Sometimes this is due to a URL firewall misconfiguration, but often the URL is either malformed or deliberately disallowed. The methods illustrated in this whitepaper will aid in determining the cause.REFERENCESNOTE:1268104.1 - Gone Access to the Requested URI has been Blocked by the URL Firewall Error Accessing IrcEmpVisitor.jspNOTE:1109537.1 - Logout Link In E-Business Suite Gets '404-PageNotFound' or '410-Gone' with oracsle and /OA_sHTML/ (typo URL) NOTE:230688.1 - How to Diagnose: 11i Basic Apache/mod_jserv Troubleshooting with Hello.class NOTE:270519.1 - Customizing an AutoConfig Environment NOTE:370721.1 - Access To The Requested Uniform Resource Identifier (URI) Has Been Blocked By The Uniform Resource Locator (URL) Firewall NOTE:380490.1 - Oracle E-Business Suite R12 Configuration in a DMZ NOTE:165195.1 - Using AutoConfig to Manage System Configurations with Oracle Applications 11i NOTE:287176.1 - DMZ Configuration with Oracle E-Business Suite 11i NOTE:364439.1 - Tips and Queries for Troubleshooting Advanced Topologies |
No comments:
Post a Comment