File/Directory Permissions Oddities After Applying Adobe ColdFusion 11 Hotfix(es)

I recently ran into an issue applying Adobe ColdFusion 11 Hotfix 7 and decided it was worth documenting for my own retention and on the off-chance that this might help someone else who bumps into it in the future. The bottom line is that at least the installers for ACF 11 Hotfixes 6 and 7 fiddle with file and folder permissions in a way that may cause problems with applying subsequent hotfixes and that seem dodgy to me.

I recently ran into an issue applying Adobe ColdFusion 11 Hotfix 7 (released 2015-11-17) and decided it was worth documenting for my own retention and on the off-chance this might help someone else who bumps into this in the future. The bottom line is that at least the installers for ACF11 Hotfixes 6 and 7 fiddle with file and folder permissions in a way that may cause problems with applying subsequent hotfixes and that seems dodgy. I emphasize “at least” because those are the only two ACF11 hotfix installers I’ve applied (as we just recently migrated from ACF10 to ACF11) and it is possible this behavior has been present in previous ACF11 hotfix installers. This behavior does not seem to be present in ACF10 hotfix installers to date. Further, I emphasize that this might cause problems because it depends on how your ACF11 is deployed and how you apply hotfixes to that deployment.

Background

I develop almost exclusively on Mac OS X, but the issue I bumped into could also be present on Linux installations. (It seems less likely to me that it will be present for Windows installations.) On my development systems, I deploy all of my application servers via WAR files as Tomcat instances on a stock Tomcat installation in a folder structure within my own user account’s home directory. My user account owns all of the files and folders within that directory structure. Further, I run, stop, and start the Tomcat instances as myself (e.g., they run as my user account and thus have no elevated privileges). None of these Tomcat instances start or stop unless I manually start or stop them, so they are only running when I need them. This approach makes it very straightforward for me to have multiple application servers (or versions of a given application server) deployed and even running concurrently if I manage ports correctly. This has been a real boon in migrating between versions of ACF, testing Railo and Lucee next to ACF, testing hotfixes… and figuring out what was going on with this particular issue. My deployment folder structure looks like this:

~/opt/t7i (all Tomcat 7 instances)
   ./cf10/ (Adobe ColdFusion 10)
   ./cf11/ (Adobe ColdFusion 11)
   ./l4/ (Lucee 4.x)
   ./l5/ (Lucee 5)
   ./railo/ (Old Railo deployment)
   ...

~/opt/t8i/ (all Tomcat 8 instances)
   ./l4/ (Lucee 4.x)
   ...

When Adobe releases ColdFusion hotfixes, I download them (typically from within the ColdFusion administrator UI), stop the relevant Tomcat instance, and then apply the hotfix from the command line, per the following example:

$ cd ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/hf-updates
$ java -jar ./hotfix-007.jar -i GUI

The Symptoms

The issue showed up when I first tried to apply ACF11 Hotfix 7 on the first of my development systems: installation of the hotfix failed with a fatal error, indicating in the installation log that it could not move files into folder ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/lib. That seemed odd, as I had previously used this same approach with multiple ACF10 hotfixes and with ACF11 Hotfix 6 on each of my development systems (including this one).

Given that my user account owns all of the files and folders where the Tomcat instance for ACF11 is installed, and that I was applying the hotfix as myself, why would working with those files and folders fail?

I reached out to the ACF team via a comment in the blog post announcing the hotfix and Nimit Sharma (@nimsharm) of the ACF team followed up with me shortly thereafter. He suggested trying to apply the hotfix as “root” (i.e., with elevated privileges) to see if that resolved the problem. After expressing my reluctance (there are reasons behind why I install and run these application servers in the manner I do), I set aside copies of the instance, and used sudo to apply the hotfix. It applied successfully.

Hmmm… what’s going on here? Clearly something related to permissions…

The Issue

After a couple run-throughs of installing ACF11 and applying first Hotfix 6 and then Hotfix 7 while keeping an eye on file ownership and permissions, I’ve determined that at least the installers for these two hotfixes fiddle with file and folder permissions within the ACF deployment in ways that do not make sense to me:

  1. The Hotfix 6 installer changes group permissions on the following files and folders from r-x to rwx:
    • ~/opt/t7i/cf11/webapps/ROOT
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/hf-updates/hf-11-00006/uninstall/.com.zerog.registry.xml
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/hf-updates/updates.xml
  2. The Hotfix 6 installer changes user folder permissions on the following folders from rwx to r-x:
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/lib
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/lib/updates

It is the removal of user-write permissions by the Hotfix 6 installer that causes installation of Hotfix 7 to fail, but adding group-write permissions on the folders is pretty suspect from a security perspective.

I’ve posed the question to Nimit of the Adobe team as to why the installers do this, but have not received a response from him so far. I can’t come up with any reason that the installer should be fiddling with group permissions on those folders (but particularly in opening up write permissions on these folders). There may be a good reason that user-write permission is removed from the other folders but it causes problems with applying subsequent hotfixes. The “Adobe ColdFusion 11 Lockdown Guide” alludes to this  (see p. 45) in discussing ownership and permissions on Linux systems and the potential for these causing problems  if not configured in a way that’s consistent with how ColdFusion is run and hotfixes are applied.

By walking through installing ACF11 (which left the folders as expected), applying Hotfix 6 (which left the needed folders unwriteable), and then applying Hotfix 7 (which failed), I’ve determined that this was not a problem with just Hotfix 6. It is a problem with both Hotfix 6 and Hotfix 7: Hotfix 7 behaves the same as Hotfix 6 in adding group-write permission to the same folders, and removes user-write permission from the same/corresponding folders.

The Resolution

Resolving the problem after applying each of the hotfixes is straightforward:

$ cd ~/opt/t7i/cf11/webapps
$ find . -perm +g+w -exec chmod -v g-w {} \;
$ find . -not -perm +u+w -exec chmod -v u+w {} \;

The second command finds any files or folders that are group-writeable and removes group-write permission. The third command finds any files or folders that are not user-writeable and adds user-write permission.

Wrapping Up

At this point, I’ve figured out why Hotfix 7 failed to install and figured out how to revert file/folder ownership and permissions to a state that makes sense. Part of my approach to installing and updating ACF my various systems now includes an additional step to find and fix any files/folders whose user/group permissions are no longer consistent.

My questions, however, remain:

  1. Why is the hotfix installer adding group-write permissions to several files/folders as part of the applying the hotfix?
  2. Why is the hotfix installer removing user-write permissions from several key folders within the ColdFusion folder structure as part of applying the hotfix, particularly when doing so leaves the ColdFusion installation in a state where future hotfixes will fail to install?

I will update this when Nimit from the Adobe team follows up with me.

Update 2016-05-14: No responses received (yet?) from anyone at Adobe, and based on my preliminary investigation with the installer for CF11 Hotfix 8 (release earlier this week), its installer behaves identically. My two questions above remain unanswered…

New Dev Boxes

It is time: I’m standing up a couple of new development systems, so I am using Tomcat 8 and Java 8 as the basis for deploying Railo and Adobe ColdFusion 10. It started as smooth sailing, but I ended up dropping back to Tomcat 7 and Java 8.

It is time: I’m standing up a couple of new development systems, so I am going to use Tomcat 8 and Java 8 as the basis for deploying Railo and Adobe ColdFusion 10 on them. This is the first time I’ve done anything with either Tomcat 8 or Java 8 as the base for the application server stack but with Java 7 set for EOL this (I first wrote “next”) year, the timing for moving seemed appropriate.

I have Railo deployed and running without issue on the first of the two systems, and have encountered no issues at all to this point.

I will blog a bit in the near future to describe my approach and a couple of minor Tomcat differences I have found.

Updated 2014-01-02: Well, that didn’t take long to go sideways… I spent several hours this afternoon trying to deploy ColdFusion 10 on Tomcat 8, and so far have been unsuccessful. I can get the WAR to unpack, but the application simply refuses to start with a very non-specific error in the Catalina log file that (I’m guessing here) appears to indicate an incompatibility with Tomcat 8 in ColdFusion 10. More to come, but my next run at this will probably be to drop back a step or two and try deploying on Tomcat 7 and Java 8.

Updated 2014-01-03: Moving forward again. As of this morning, I now have both Railo 4 and ColdFusion 10 deployed against Tomcat 7 and Java 8.

A Gotcha: Adobe ColdFusion 10, Apache Tomcat, J2EE Sessions

Enabling J2EE sessions with Adobe ColdFusion (ACF) on Apache Tomcat also requires disabling session persistence in Tomcat… and it took me a while to figure that out. I learned quite a bit about just how flexible Tomcat is and I am kicking myself for not exploring ACF on stock Tomcat a long time ago.

I’ve spent a bit of time over the past month or so playing with setting up a couple of my development systems to run both Railo and Adobe ColdFusion 10 concurrently on a stock install of Apache Tomcat 7. In the case of ACF10, my interest in running on stock Tomcat is based on a desire get away from the custom-built and now out-dated version of Tomcat which Adobe (unwisely, in my opinion) bundles with ACF10. This past week I bumped into a bit of a gotcha that took me quite a bit of time to track down and solve. In the hopes of helping others avoid this same problem and so that I don’t forget it, I’ll share what I ran into and how to solve it.

The symptoms I was seeing were that as soon as I enabled J2EE sessions in the CF administrator and then subsequently stopped Tomcat for any reason, Tomcat would no longer start cleanly, it was no longer bringing up the context running ACF, and because it was not starting cleanly it would also not shutdown cleanly. As weird as this seemed (and sounds, I realize), this was very repeatable. I had it down to something I could reproduce in under 5 minutes: set up a clean new Tomcat install, deploy ACF10 on it via a WAR file, sign into the CF admin, enable J2EE sessions, stop Tomcat… and it would no longer start. There was nothing in the ${CATALINA_BASE}/logs/catalina.out log file indicating what was wrong; it just looked like Tomcat would hang as it was starting. I determined I could install, deploy, and then start/stop Tomcat successfully as many times as I wanted and the problem would not show up until and unless I enabled J2EE sessions in the CF admin. It did not seem to be dependent on whether ACF10 was patched or the specific version of Tomcat.

At one point, I even dug around and figured out how to change the log level used by Catalina, in the hopes that a bit more detail might shed some light on the problem but the bump of one additional level of detail took the log entries on starting Tomcat from under 40 to over 15,000… and while the nature of the problem — in retrospect, of course — might well have been touched on in that blizzard of log entries, I couldn’t find anything even remotely resembling a needle in that sea of haystacks.

It seemed like something specifically related to turning on J2EE sessions in ACF10 was breaking Tomcat. Based on a suggestion from a co-worker, I removed write permissions for the user under which Tomcat was running from all folders under ${CATALINA_BASE} except the ./logs/ and ./webapps/ folders, in an attempt to see if I could determine where the breakage was occurring. On starting Tomcat, I noticed a complaint in the ${CATALINA_BASE}/logs/catalina.out log file about not being able to write to folder ${CATALINA_BASE}/work/Catalina/localhost/_. Looking in that folder, I found a file named SESSIONS.ser. Doing a bit of Googling, I came across a short blog post dealing with session persistence across Tomcat restarts.

I’m not going to pretend that I know why Tomcat would have session persistence enabled across restarts, or why I might want to persist sessions across restarts (I really can’t come up with a scenario where I would want that), or why enabling J2EE sessions in ACF10 would seem to break this persistence… but clearly it does. That file gets written on stopping Tomcat and is then read and removed when Tomcat next starts/restarts. To disable this persistence, the context(s) within Tomcat on which ACF10 is enabled need to include a Tomcat session manager component specifically configured to disable this persistence:

...
<Manager pathname="" />
...

This is touched upon in the Apache Tomcat docs here. That session manager component can occur in any of the supported locations where contexts are configured in Tomcat. Once I confirmed this to solve the problem, I took a quick peek at the context configuration for ACF10 when it is running against the Adobe-provided custom build of Tomcat 7.0.23, and — sure enough — it disables session persistence in precisely this same manner.

The one upside to all of the time I spent tracking this down is that I can now install Tomcat and deploy ACF on it in a matter of just a few minutes in a variety of ways, including multiple virtual hosts under a single Tomcat, multiple Tomcat installs running on different ports, and  a single Tomcat install with multiple instances via use of ${CATALINA_HOME} and ${CATALINA_BASE}. I’m kicking myself for not looking at deploying ACF on stock Tomcat three years ago when I learned of Adobe’s choice with ACF10 to use a non-stock Tomcat that they inexplicably have not updated.

Tomcat: An IPv6 Address Filtering Gotcha

In which we look at a small gotcha related to IPv6 addresses and how that gotcha has to be accounted for as we allow/deny requests to our Tomcat server via IPv6-based URIs.

This is another in my series of short posts covering my efforts to stand up a Tomcat/ACF10 development environment. For background, see the first post in the series.

In my previous post, I revisited allowing access to pages served up by Tomcat to just requests originating from the local system. In that post, we looked at use of Tomcat’s Remote Address Filter in order to have any denied requests be provided with a custom error page. As I noted, however, when I tried to access my Tomcat server via an IPv6-based URI (e.g., http://[::1]:8501/test.cfm), my request was being denied even though I had configured the filter to allow access to requests from the following addresses (specified via regex):

  • 127\.\d+\.\d+\.\d+
  • ::1
  • 0:0:0:0:0:0:0:1

Given the second and third of those address specifications, I would have expected my request to have been allowed. This is where having access logs configured comes in handy. Those logs showed these requests coming from 0:0:0:0:0:0:0:1%0 and being denied. OK… what’s with that trailing “%0”?

A bit of digging and I had my answer: the IPv6 address specification includes something called “scopes” that may be present. That “%0” is the default scope and (as the default) is optional. The RFC for these scopes is pretty fuzzy, so for now will allow for an optional non-negative decimal integer for the scope (per the RFC) and if we ever end up with something else (which the RFC indicates is possible) we will revisit this. We will update our regexes to account for the possible presence of the scope:

  • 127\.\d+\.\d+\.\d+
  • ::1(%\d+)?
  • 0:0:0:0:0:0:0:1(%\d+)?

Update the list of address regexes in the filter definition, bounce the server, and Tomcat should now allow requests when invoked with an IPv6-based URI. Now we can get back to looking at the extent to which ACF10 supports IPv6.

A final thought on this: you may or may not encounter this, as I have the distinct impression this behavior is a function of the environment (at least the OS and/or the JDK under Tomcat). I’ve blogged it simply because I tripped over it.

Tomcat: Revisiting IP Access Restriction

Another in my series of short posts looking at Tomcat configuration as I set up a Tomcat/ColdFusion10 development server environment, in which we take a second look at restricting access to the server based on the IP address from which the request originated.

This is another in my series of short posts on configuring Tomcat as I get a Tomcat/ACF10 development environment configured. For a bit of background, see the first post in the series.

We took a quick look earlier at restricting access to my Tomcat server based on the IP address from which the request originated. In that first take, we used Tomcat’s Remote Address Valve component to restrict access to just requests from the local system. In that post, I indicated we would probably look at this subject again in the future. Here’s my second look… and I know we have at least one more coming.

One of the first items I plan on digging into with ACF10 is how it deals with IPv6. As I started poking around the edges of this, I discovered two things that merited further investigation:

  1. Requests to my local server with an IPv6-based URI (e.g., http://[::1]:8501/sample.cfm) were being blocked despite the inclusion of the IPv6 versions of the local server address (i.e., ::1 and 0:0:0:0:0:0:0:1) in our earlier Valve-based set of allowed addresses, and
  2. I was not seeing my custom 403 error page that I expected, based on my earlier work to configure one

To figure out what’s going on with the second of those two items, let’s look at where the addressed-based restrictions were implemented vs. where the custom error pages were configured:

  • My Valve-based address restrictions were implemented within the Host element in ./cfusion/runtime/conf/server.xml
  • My custom error pages were configured in the web-app element within ./WEB-INF/web.xml

It’s not too big a leap to see that the two items are configured at two different levels within the overall configuration of Tomcat and the ColdFusion Web application. This is one of those places I’ve bumped into the “there is more than one way to get there” aspect of Tomcat. Tomcat also provides a Remote Address Filter which provides the same functionality as the valve we used in our first run at this. As a filter, it is configured for the Web application (i.e., at the same level as our custom error pages) so it made sense this might address problem #2 from above.

Add the following filter specification to ./WEB-INF/web.xml as the first filter:

<filter>
   <filter-name>Remote Address Filter</filter-name>
   <filter-class>org.apache.catalina.filters.RemoteAddrFilter</filter-class>
   <init-param>
      <param-name>allow</param-name>
      <param-value>127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1</param-value>
   </init-param>
</filter>
...

Add the following filter-mapping specification to the same file as the first such mapping:

<filter-mapping>
   <filter-name>Remote Address Filter</filter-name>
   <url-pattern>/*</url-pattern>
</filter-mapping>
...

Remove the corresponding Valve specification from ./cfusion/runtime/conf/server.xml. Bounce the server and verify that the custom error pages are still being served appropriately.

In my next post, we’ll figure out what’s going on with IPv6-based URI’s and why we’re still being blocked (although now we at least get our custom error page). And as noted above, we’ll revisit this whole subject of address-based access restrictions at least one more time in the future.