Configuration belongs to the environment and not to the application

Sometime ago I helped a client create an internal .net web application. One of the decisions that we had to make was on having a different web.config file for different environments. On a previous project, I had used xml poke nant tasks to do it. It had worked for us then, so I solved the problem in the same way here. This project was greenfield and we decided to go with Powershell and Psake (nant is xml and xml isn’t the best language for a deployment script). So we rolled out our own configuration manager. It took us some time to write, but we somehow managed to do it in Powershell. Each environment had its own config.xml file which contained environment specific properties. Our deployment script would just poke the web.config (which was dev config by default) with environment specific values. It kinda worked on the QA and UAT environment. However when we were supposed to deploy it to production, the ops guys had a policy where a sensitive thing such as a database password could not stay in the app folder and would be kept in the default .net config folder (which was locked down). Fair requirement, but our configuration management script would just break because even if the database password would be in the default .net directory, it would get overridden by the password in the app folder (because it was packaged with the code). This meant that for production, we had to delete the web.config file in the app directory and then the ops guys had to put in their version of the web.config in the default config folder. This approach had certain problems; for one the app structure and the deployment script were different in production and other environments. Another issue was that we had to manually give a new version of the web.config to the ops guys when we added or removed a section in the web.config.

This was before I had read the Continuous Delivery book which describes the same problem and suggests that you keep the code and configuration separate. I also worked on a project where we used Puppet extensively to manage configuration. That is when it struck me, that a lot (even all ?) of configuration belongs to the environment and not to the application. Keeping it tied to the application just makes it a maintenance nightmare. The other lesson I learnt was that we reinvented the wheel by writing our own configuration manager. Tools like Puppet and Chef already have solved the problem. You don’t need xml poke anymore, just a template with variables that vary across environment. You can have the passwords and other data stored in key value pairs or hierarchically (environment wise) in an external repository and just pull them when required. Production can have a different secure repository. There is a learning curve with the tools however, but it is definitely worth the time.

As developers we spend most of our time on dev and test environments, therefore we don’t see requirements that are very obvious to the ops guys. We design ‘elegant’ solutions where each ‘app owns it’s configuration’. From an automated deployment and continuous delivery perspective, this approach becomes a pain when the requirements of your environment change. A good approach is to have your dev configuration separately managed in exactly the same way as it would be in production by the same configuration management tool. I will never do xml poke ever again; after working with a configuration management tool it just feels like a hack.

Advertisements

5 comments

  1. Great topic! I’m curious for your answers on a few questions:

    How do you manage configuration in development? and how do you coordinate the process of developers writing code that relies on new configurations? How do you keep developers in sync with your operations team in regard to configurations?

  2. A key principle to keep in mind is to keep app code and configuration separate. Even in separate version control repositories. The config repo is also code, but in the puppet/chef DSLs. It is shared between the ops and the dev teams and is common for all environments (including the dev and production environment). These tools also make sure that the same version software also gets installed on all environments (more predictability). So when a dev makes a config change, the differences in requirements for every environment are very visible to the dev because it is the same codebase for all the environments. Even if the dev screws up something, the version control system gives you the comfort of reverting the change. The shared repo is a great implicit collaboration tool between the dev and ops team. Of course all this requires a certain level of trust, rapport and communication between the dev and ops camps. Thats what the whole devops movement is about isn’t it. 🙂

  3. Pingback: IT Automation Digest for January 28, 2013 | Puppet Labs

  4. My question is similar to Ben’s. You mention having a template and pulling variables to fill in that template. What do you use to handle that? Do you just use the templating that’s built into puppet/chef or do you use some other program to do it? I ask because right now at where I work we have this custom java app that will go and replace template variables in various config files. But it’s a kludge and everyone aside from the guy that made it seems to swear at it all the time. I’ve tried to think of something that exists already thought and couldn’t come up with anything. Ruby has the concept of template files. Could run sed on some template files, but that’s all still a custom thing that there should already be an answer to.

    • The DSLs of both puppet and chef give you templates. They are something like jsp files/ruby templates (in fact puppet uses the erb syntax) with placeholders for variables. You can have a template for each config file that your application needs. How you pass in those variables depends on how you design your puppet manifests/chef recipes (which is code). The anti pattern is to write your configuration manager all over again when a new app comes (like the one you have in java right now). Configuration management tools give you a DSL which is designed for solving configuration issues, all you have to do is concentrate on getting the right variables into the templates. The configuration management system will substitute the variables (for each environment) and make sure that the config file gets placed in the right location.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s