File/Directory Permissions Oddities After Applying Adobe ColdFusion 11 Hotfix(es)

I recently ran into an issue applying Adobe ColdFusion 11 Hotfix 7 (released 2015-11-17) and decided it was worth documenting for my own retention and on the off-chance this might help someone else who bumps into this in the future. The bottom line is that at least the installers for ACF11 Hotfixes 6 and 7 fiddle with file and folder permissions in a way that may cause problems with applying subsequent hotfixes and that seems dodgy. I emphasize “at least” because those are the only two ACF11 hotfix installers I’ve applied (as we just recently migrated from ACF10 to ACF11) and it is possible this behavior has been present in previous ACF11 hotfix installers. This behavior does not seem to be present in ACF10 hotfix installers to date. Further, I emphasize that this might cause problems because it depends on how your ACF11 is deployed and how you apply hotfixes to that deployment.

Background

I develop almost exclusively on Mac OS X, but the issue I bumped into could also be present on Linux installations. (It seems less likely to me that it will be present for Windows installations.) On my development systems, I deploy all of my application servers via WAR files as Tomcat instances on a stock Tomcat installation in a folder structure within my own user account’s home directory. My user account owns all of the files and folders within that directory structure. Further, I run, stop, and start the Tomcat instances as myself (e.g., they run as my user account and thus have no elevated privileges). None of these Tomcat instances start or stop unless I manually start or stop them, so they are only running when I need them. This approach makes it very straightforward for me to have multiple application servers (or versions of a given application server) deployed and even running concurrently if I manage ports correctly. This has been a real boon in migrating between versions of ACF, testing Railo and Lucee next to ACF, testing hotfixes… and figuring out what was going on with this particular issue. My deployment folder structure looks like this:

~/opt/t7i (all Tomcat 7 instances)
   ./cf10/ (Adobe ColdFusion 10)
   ./cf11/ (Adobe ColdFusion 11)
   ./l4/ (Lucee 4.x)
   ./l5/ (Lucee 5)
   ./railo/ (Old Railo deployment)
   ...

~/opt/t8i/ (all Tomcat 8 instances)
   ./l4/ (Lucee 4.x)
   ...

When Adobe releases ColdFusion hotfixes, I download them (typically from within the ColdFusion administrator UI), stop the relevant Tomcat instance, and then apply the hotfix from the command line, per the following example:

$ cd ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/hf-updates
$ java -jar ./hotfix-007.jar -i GUI

The Symptoms

The issue showed up when I first tried to apply ACF11 Hotfix 7 on the first of my development systems: installation of the hotfix failed with a fatal error, indicating in the installation log that it could not move files into folder ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/lib. That seemed odd, as I had previously used this same approach with multiple ACF10 hotfixes and with ACF11 Hotfix 6 on each of my development systems (including this one).

Given that my user account owns all of the files and folders where the Tomcat instance for ACF11 is installed, and that I was applying the hotfix as myself, why would working with those files and folders fail?

I reached out to the ACF team via a comment in the blog post announcing the hotfix and Nimit Sharma (@nimsharm) of the ACF team followed up with me shortly thereafter. He suggested trying to apply the hotfix as “root” (i.e., with elevated privileges) to see if that resolved the problem. After expressing my reluctance (there are reasons behind why I install and run these application servers in the manner I do), I set aside copies of the instance, and used sudo to apply the hotfix. It applied successfully.

Hmmm… what’s going on here? Clearly something related to permissions…

The Issue

After a couple run-throughs of installing ACF11 and applying first Hotfix 6 and then Hotfix 7 while keeping an eye on file ownership and permissions, I’ve determined that at least the installers for these two hotfixes fiddle with file and folder permissions within the ACF deployment in ways that do not make sense to me:

  1. The Hotfix 6 installer changes group permissions on the following files and folders from r-x to rwx:
    • ~/opt/t7i/cf11/webapps/ROOT
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/hf-updates/hf-11-00006/uninstall/.com.zerog.registry.xml
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/hf-updates/updates.xml
  2. The Hotfix 6 installer changes user folder permissions on the following folders from rwx to r-x:
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/lib
    • ~/opt/t7i/cf11/webapps/ROOT/WEB-INF/cfusion/lib/updates

It is the removal of user-write permissions by the Hotfix 6 installer that causes installation of Hotfix 7 to fail, but adding group-write permissions on the folders is pretty suspect from a security perspective.

I’ve posed the question to Nimit of the Adobe team as to why the installers do this, but have not received a response from him so far. I can’t come up with any reason that the installer should be fiddling with group permissions on those folders (but particularly in opening up write permissions on these folders). There may be a good reason that user-write permission is removed from the other folders but it causes problems with applying subsequent hotfixes. The “Adobe ColdFusion 11 Lockdown Guide” alludes to this  (see p. 45) in discussing ownership and permissions on Linux systems and the potential for these causing problems  if not configured in a way that’s consistent with how ColdFusion is run and hotfixes are applied.

By walking through installing ACF11 (which left the folders as expected), applying Hotfix 6 (which left the needed folders unwriteable), and then applying Hotfix 7 (which failed), I’ve determined that this was not a problem with just Hotfix 6. It is a problem with both Hotfix 6 and Hotfix 7: Hotfix 7 behaves the same as Hotfix 6 in adding group-write permission to the same folders, and removes user-write permission from the same/corresponding folders.

The Resolution

Resolving the problem after applying each of the hotfixes is straightforward:

$ cd ~/opt/t7i/cf11/webapps
$ find . -perm +g+w -exec chmod -v g-w {} \;
$ find . -not -perm +u+w -exec chmod -v u+w {} \;

The second command finds any files or folders that are group-writeable and removes group-write permission. The third command finds any files or folders that are not user-writeable and adds user-write permission.

Wrapping Up

At this point, I’ve figured out why Hotfix 7 failed to install and figured out how to revert file/folder ownership and permissions to a state that makes sense. Part of my approach to installing and updating ACF my various systems now includes an additional step to find and fix any files/folders whose user/group permissions are no longer consistent.

My questions, however, remain:

  1. Why is the hotfix installer adding group-write permissions to several files/folders as part of the applying the hotfix?
  2. Why is the hotfix installer removing user-write permissions from several key folders within the ColdFusion folder structure as part of applying the hotfix, particularly when doing so leaves the ColdFusion installation in a state where future hotfixes will fail to install?

I will update this when Nimit from the Adobe team follows up with me.

Update 2016-05-14: No responses received (yet?) from anyone at Adobe, and based on my preliminary investigation with the installer for CF11 Hotfix 8 (release earlier this week), its installer behaves identically. My two questions above remain unanswered…

Setting Up a Fake SMTP Server

On my team’s various development boxes (which are almost always laptops and therefore not always connected to our corporate intranet and sometime not on any network), we have typically configured our ColdFusion and Railo application servers to use our internal mail servers. Many of our Web applications send user-composed, automated, or process-based email messages, in addition to security-related event notifications. We’ve been bitten a couple of times in developing this mail-related logic, with messages being generated and sent to real system users from our development servers which, as you can imagine, has significant potential for generating concern and/or confusion on the users’ part. This hasn’t happened recently, in part because we are collectively more careful and in part based on our approach to how email gets addressed depending on whether the app is running on a development, testing, or production server.

I’ve long believed that a safer approach would be to configure our non-production servers to use something other than a “real” mail server. I’m in the process of standing up a new development system for myself, and decided I was going to poke at this a bit. For this first foray, I wanted something

  • simple to set up, configure, start, and stop
  • free
  • preferably available cross-platform (my team develops on Mac OS, Linux, and Windows systems based on developer preference for platform)
  • completely local (because of the need to be able to run without being on a network)

A solution would need to support SMTP, as both the ColdFusion and Railo application servers are configured to talk to an SMTP server. Anything beyond that set of basic requirements would be a bonus.

After a bit of looking, I decided to use FakeSMTP. It ticks all of the above boxes, and is implemented as a single Java JAR compatible with Java 1.6 and above. It has a simple UI for watching activity and  “outbound” mail handed to it, and can easily be configured to store transmitted mail messages to disk as .eml files for review (making developing email-related content fairly straightforward).

I use a bash shell script to start and stop my Tomcat instances for ColdFusion and Railo, so I added logic to that script to optionally start and stop my local mail server, as follows (wrapped here only for presentation):

# Start our fake SMTP server:
if [ "${3}" = "mail" ]; then
  java -jar ~/opt/fakesmtp/fakesmtp-1.13.jar -s -b -p 2525 \
    -o ~/opt/fakesmtp/out/ -a 127.0.0.1 \
    >> ~/opt/fakesmtp/logs/fakesmtp-`date +%Y%m%d`.log &
fi

and

# Stop our fake SMTP server:
if [ "${3}" = "mail" ]; then
  kill `ps -a > /tmp/ps.out && grep -i fakesmtp /tmp/ps.out | \
    cut -f 2 -d ' ' -`
fi

The start logic runs the mail server in the background (as I really don’t intend to use the UI), listening on address 127.0.0.1 and port 2525 (to avoid needing to run with elevated privileges on the default SMTP port of 25). It dumps mail messages to a folder for review, and logging server activity to a log file with a date-based name. The stop logic finds the mail server process and kills it.

I’ve tested it with both ColdFusion and Railo application servers, and it works. I will probably add some shutdown-related logic to do a bit of cleanup to make sure neither the log files or the saved email files get out of control.

Final thought: If it weren’t for my desire to be able to run completely standalone at times where I either do not have access to or do not want to use a network connection, I probably would have gone with something like Debug Mail. I have not used it, but it looks excellent. Given Railo’s ability to configure multiple mail servers, I may still play with it a bit as part of my solution.

New Dev Boxes

It is time: I’m standing up a couple of new development systems, so I am going to use Tomcat 8 and Java 8 as the basis for deploying Railo and Adobe ColdFusion 10 on them. This is the first time I’ve done anything with either Tomcat 8 or Java 8 as the base for the application server stack but with Java 7 set for EOL this (I first wrote “next”) year, the timing for moving seemed appropriate.

I have Railo deployed and running without issue on the first of the two systems, and have encountered no issues at all to this point.

I will blog a bit in the near future to describe my approach and a couple of minor Tomcat differences I have found.

Updated 2014-01-02: Well, that didn’t take long to go sideways… I spent several hours this afternoon trying to deploy ColdFusion 10 on Tomcat 8, and so far have been unsuccessful. I can get the WAR to unpack, but the application simply refuses to start with a very non-specific error in the Catalina log file that (I’m guessing here) appears to indicate an incompatibility with Tomcat 8 in ColdFusion 10. More to come, but my next run at this will probably be to drop back a step or two and try deploying on Tomcat 7 and Java 8.

Updated 2014-01-03: Moving forward again. As of this morning, I now have both Railo 4 and ColdFusion 10 deployed against Tomcat 7 and Java 8.

Adobe ColdFusion Docs: A better way?

I’m not a fan of the structure the Adobe ColdFusion (ACF) team has moved to for documentation for the recent versions of their product.

A bit of background

With the recent (late May 2014) release of ACFv11, there are now three versions of ACF in which I have varying degrees of interest:

  • v9: we are in the final stages of moving our last app from a shared v9 server to one of our dedicated v10 servers,
  • v10: our entire dev environment is based on v10, and all of our apps with the noted exception above run on v10 in production, and
  • v11: the recent release we’ve watched from a distance as we weigh whether to continue with ACF in the future or shift to Railo as our CFML engine.

My team has used ColdFusion since its Allaire v4 days, and have historically had multiple versions in play across our development and production environments, so the above situation covering three versions is very typical for us. In fact, having all of our dev and production environments based on a single version of ACF as we (almost) do now represents the first time in at least 10 years we’ve been able to achieve this. I don’t believe we are unique in this: any developer moving between versions of the product (or even considering such a migration) will be working with at least two versions of the language and/or the documentation before and during their migration effort.

In addition, I built and continue to maintain a CFML language mode for ActiveState’s Komodo IDE and Edit editors, providing syntax highlighting and tag/tag attribute completion. The tag/tag attribute completion is specific to each CFML version, allowing the user to designate which version of the language is to be used as the basis for which tags and which attributes are to be provided as options to the user. I rely heavily on version-specific documentation to build each of these version-specific implementations.

The problem with the current wiki approach

In recent versions of ACF up through v9, each version had comprehensive documentation sets available both online and as downloadable PDFs. Each of these documentation sets were separate and distinct, and the online versions made it very straightforward to shift between versions. With v10, there is a downloadable PDF version of the documents (thanks to Adam Cameron for pointing me toward those) but the only references to it are not particularly easy to find (search the wiki for “archive” — how obvious is that if you are looking for v10 documentation?). Googling for “adobe coldfusion 10 documentation” does not yield that set of documents in the first page of results. Further, there does not seem to be an online version of some portions of the v10 (such as the tag/function reference). The current online version of the documentation is maintained as part of a wiki that does not contain discrete sections for different versions of the product or language, and for the most part appears to be focused on v11.

The old version-specific implementation of the online docs has the tremendous benefit to the user of being crystal clear as to what capabilities are available in a given version of the language (e.g., which attributes are available for a given tag, what functions/function arguments are available). If I am working on an app currently running on version “x” of ACF, I typically want to see only the version “x” documentation for that tag.

The current wiki-based documentation set, however, does not have that clear distinction between versions of the product or language. This makes differentiating between versions for tags, attributes, supported attribute values, functions, etc., in terms of what is valid and supported between versions far more challenging and error-prone than in previous versions. If I care about the v10 implementation of the cfzip tag, for instance, I have to either use the off-line version of the docs or make sure I look at the “history” portion of the relevant tag page and then mentally remove the new attributes added for v11 (“password” and “encryptionalgorithm”, in this particular case) as I scan down through the attribute documentation — particularly given that there is nothing within the description of the individual attributes indicating their recent addition in v11 (and in the case of the “password” attribute, it seems to be missing altogether from the attribute list).

A better approach?

It seems to me that a better approach would be for separate, discrete version-specific sections within the wiki. The language reference — along with the supporting documents identifying additions, removals, and deprecations — is one example of where this would make the documentation much easier to use. The v10 portion of the reference would always be specific to v10 and would not (should not?) need to contain any v11 content. The v11 portion of the reference would presumably start as a clone of the v10 portion of the documentation, and could evolve along with the language as v11 was developed. As work on v12 is started, the v11 portion would be cloned as a draft and evolve independently from the sections covering previous versions.

The current wiki approach is likely to get more and more unwieldy as additional versions of the product and the language are covered within the current wiki structure. Take the page listing deprecations and removals as an example. Besides being in drastic need of updates as of this writing for completeness and currency with v11, this page — based on its current structure — will get more and more unwieldy as additional versions have to be covered: more rows to cover features being deprecated or removed, and more columns for subsequent versions? The current approach just does not seem to scale for pages such as this.

Note that I am not taking issue with the use of a CMS or wiki for the documentation itself as much as what I see as a poor decision of how to structure the documentation within the tool the ACF team selected.

Finding documentation for earlier, but still-supported, versions of a product should not be difficult but in this case it is getting harder. I’d like to believe the ACF documentation team will recognize this and restructure the documentation set before it collapses under its own weight and degenerates even further in terms of being usable and useful.

And the off-line docs?

The current wiki structure — which seems to be focused mostly on the current language version — also raises the question for me as to whether a similar set of downloadable v11 references will be made available now that the product is available or as the wiki gets refocused on v12 (presumably in the near future, immediately after all of the existing v9/10/11 bugs have been resolved). That’s a separate but important and relevant question. I hope they do continue making that format available, as those downloadable PDFs have repeatedly proven valuable for me and my team at various times.

A Gotcha: Adobe ColdFusion 10, Apache Tomcat, J2EE Sessions

I’ve spent a bit of time over the past month or so playing with setting up a couple of my development systems to run both Railo and Adobe ColdFusion 10 concurrently on a stock install of Apache Tomcat 7. In the case of ACF10, my interest in running on stock Tomcat is based on a desire get away from the custom-built and now out-dated version of Tomcat which Adobe (unwisely, in my opinion) bundles with ACF10. This past week I bumped into a bit of a gotcha that took me quite a bit of time to track down and solve. In the hopes of helping others avoid this same problem and so that I don’t forget it, I’ll share what I ran into and how to solve it.

The symptoms I was seeing were that as soon as I enabled J2EE sessions in the CF administrator and then subsequently stopped Tomcat for any reason, Tomcat would no longer start cleanly, it was no longer bringing up the context running ACF, and because it was not starting cleanly it would also not shutdown cleanly. As weird as this seemed (and sounds, I realize), this was very repeatable. I had it down to something I could reproduce in under 5 minutes: set up a clean new Tomcat install, deploy ACF10 on it via a WAR file, sign into the CF admin, enable J2EE sessions, stop Tomcat… and it would no longer start. There was nothing in the ${CATALINA_BASE}/logs/catalina.out log file indicating what was wrong; it just looked like Tomcat would hang as it was starting. I determined I could install, deploy, and then start/stop Tomcat successfully as many times as I wanted and the problem would not show up until and unless I enabled J2EE sessions in the CF admin. It did not seem to be dependent on whether ACF10 was patched or the specific version of Tomcat.

At one point, I even dug around and figured out how to change the log level used by Catalina, in the hopes that a bit more detail might shed some light on the problem but the bump of one additional level of detail took the log entries on starting Tomcat from under 40 to over 15,000… and while the nature of the problem — in retrospect, of course — might well have been touched on in that blizzard of log entries, I couldn’t find anything even remotely resembling a needle in that sea of haystacks.

It seemed like something specifically related to turning on J2EE sessions in ACF10 was breaking Tomcat. Based on a suggestion from a co-worker, I removed write permissions for the user under which Tomcat was running from all folders under ${CATALINA_BASE} except the ./logs/ and ./webapps/ folders, in an attempt to see if I could determine where the breakage was occurring. On starting Tomcat, I noticed a complaint in the ${CATALINA_BASE}/logs/catalina.out log file about not being able to write to folder ${CATALINA_BASE}/work/Catalina/localhost/_. Looking in that folder, I found a file named SESSIONS.ser. Doing a bit of Googling, I came across a short blog post dealing with session persistence across Tomcat restarts.

I’m not going to pretend that I know why Tomcat would have session persistence enabled across restarts, or why I might want to persist sessions across restarts (I really can’t come up with a scenario where I would want that), or why enabling J2EE sessions in ACF10 would seem to break this persistence… but clearly it does. That file gets written on stopping Tomcat and is then read and removed when Tomcat next starts/restarts. To disable this persistence, the context(s) within Tomcat on which ACF10 is enabled need to include a Tomcat session manager component specifically configured to disable this persistence:

...
<Manager pathname="" />
...

This is touched upon in the Apache Tomcat docs here. That session manager component can occur in any of the supported locations where contexts are configured in Tomcat. Once I confirmed this to solve the problem, I took a quick peek at the context configuration for ACF10 when it is running against the Adobe-provided custom build of Tomcat 7.0.23, and — sure enough — it disables session persistence in precisely this same manner.

The one upside to all of the time I spent tracking this down is that I can now install Tomcat and deploy ACF on it in a matter of just a few minutes in a variety of ways, including multiple virtual hosts under a single Tomcat, multiple Tomcat installs running on different ports, and  a single Tomcat install with multiple instances via use of ${CATALINA_HOME} and ${CATALINA_BASE}. I’m kicking myself for not looking at deploying ACF on stock Tomcat three years ago when I learned of Adobe’s choice with ACF10 to use a non-stock Tomcat that they inexplicably have not updated.