The more businesses and development organisations I work with, the more I see common problem “themes”. In this series of articles I am trying to highlight the problems and show the potential solutions to some of these common symptoms.

I occasionally see a tendency in teams to attempt to spike 'everything' that is unknown to get a completely clear view for estimation or build.  In the extreme this leads to a situation where every feature is spiked to some degree before being built. Clearly wasteful.

Usually this behaviour is found in teams where for some reason either trust is low or there is a fear of failure, so tendency to spike is a protection mechanism to ensure that the team know they can deliver. More often than not in these situations the result of the spike could easily have been committed and delivered, often the result being that the estimation then becomes so low as to be trivial (because the code is in the bag), just with inevitable wasted time.

So the aim is to ensure the team feel willing to work with some unknowns - or are willing to take some risk into build. That they aren't aiming to know everything before they commit to building!

So ask if the spike is really necessary - what is the risk that means we don't think we can commit? If the risk is simply that "we can't be 100% sure we can deliver" or "we have never done that before" then you don't need a spike. If the risk is that "the world may implode" or "it may cause irreparable damage to our fragile ecosystem ultimately leading to the extinction of the human race" - then you may want to consider if a spike will actually help reduce the risk.

I have been using msdeploy for years for automating ASP.NET deployments, I always knew it was a clever/cool tool, but I really had no idea of the depth!

I had a bit of an issue very recently whereby I needed to get structure and data from a SQL CE SDF database file (don't ask!!). I thought this was going to burn a morning - obviously my Google fu kicked in and I found a useful article on migrating between database formats. This pointed me to using msdeploy command line, so reading the docs to see what I could do I was surprised to see the depth – just look at al the providers available!

I ended up running a simple command line to extract a full SQL Server DDL/DML script allowing me to create a copy of the database wherever.

msdeploy -verb:sync -source:dbFullSql="Data Source={{ Path to SDF file! }}",sqlCe=true -dest:dbFullSql="{{ Path to output file}}"

Script generated and full SQL DB created in less than five minutes from opening the browser tab!

This is definitely a tool to have in your utility belt…

The more businesses and development organisations I work with, the more I see common problem “themes”. In this series of articles I am trying to highlight the problems and show the potential solutions to some of these common symptoms.

When I visit companies to help them with software development I end up asking a lot of questions - often finding myself sounding like an annoying 5 year old child just repeating the word "Why?" in response to every answer! When I get down to responses like "because we have always done it that way" or "that’s' just the way we do it" then I know the way things are being done needs to be reviewed. It's like a smell - if we haven't thought about something we are doing for so long that it becomes second nature, how can we be sure it is still valuable or relevant?

In software development when we are trying to improve something (or to generally be innovative) failures and mistakes are seen as a necessary part of the process of working out what's right. The path to the successful solution is comprised of a series of small mistakes and failures from which we learn, adapting the solution as we go. I often find that development teams neglect applying this practice to the way they build software. If we take to be fact that nobody is perfect, and that things can always be improved - then it's OK to make mistakes and it's OK to get things wrong. What's not OK is to repeat the same mistakes ad infinitum!

I like the metaphor of friction when looking at issues in the development process, friction holding us back and slowing down our delivery vehicle!

Friction can be felt anywhere from the initial idea right through to running in production, and friction may be felt in one part of the process but caused elsewhere. I think it is easiest to spot (and fixing has the greatest gain) in the things we do regularly - like deployment. Teams I have worked with have "got used" to the process of creating builds and deployment taking days. It's obviously never that hard!

No matter how you are building software this is why taking time to "inspect" what we do so that we can make small adjustments toward continuous improvement is so important. Take time to note what you do, how long it takes, what feels productive, what’s frustrating etc. Reflect over this as a team and honestly question and evaluate what you do and how. Don’t be afraid to try and make small changes, measure the effect and adjust as necessary – removing friction makes you faster. Don't just do things "'cos that's what we do" - think and reflect – it *may* still be the right way, but there may be a better way. The world doesn't stand still so nor should we!

Working as a consultant with lots of different teams helping to promote engineering practices I often hear ‘reasons’ as to why teams and developers don’t or can’t write automated tests. Most developers and teams want to do things that will ultimately make their lives easier, but sometimes hurdles and issues seem too big to overcome, and so they stop trying! One conversation with a particular developer a while back went along the lines of “the tests slow me down”. This intrigued me as personally I find that TDD speeds me up - so I asked a few more questions and paired with him to see his approach. Long story short I found that when building this developer changes his mind and his constructor injected dependencies a lot, and without auto mocking he was changing his mocks and construction more than his code.

Everyone’s approach is different – that’s what makes the world so great right – so when I listened and understood the pain I was able to offer a solution. I think all too often we just say “do this” or “do that” without looking at the reasons people may be struggling…

I will try and demonstrate the change in approach to auto mocking that helped in this case using a massively fictitious (and equally tragic) example.

We have an article service that returns articles based on a string Identifier (presumably to display somewhere). The first test written using NUnit and Moq to read the article looks like so:

 1 [Test]
 2 public void returns_read_article()
 3 {
 4     var repository = new Mock<IArticleRepository>();
 5     repository.Setup(r => r.Get(It.IsAny<string>())).Returns(new Article());
 6 
 7     var service = new ArticleService(repository.Object);
 8     
 9     var article = service.Get(Guid.NewGuid().ToString());
10     Assert.IsNotNull(article);
11 }   

And the simplest ‘production’ code looks like:

 1 public class ArticleService
 2 {
 3      private readonly IArticleRepository _repository;
 4 
 5      public ArticleService(IArticleRepository repository)
 6      {
 7          _repository = repository;
 8      }
 9 
10      public Article Get(string articleId)
11      {
12          return _repository.Get(articleId);
13      }
14 } 

I told you it was a tragic example right!

So the issue this developer had was that whenever he changed his mind and needed to introduce another dependency, all of his tests need to be changed (OK in this case it’s one test, but if you have 20 tests that’s a bit more work). Let’s simulate the change with another tragic example – let’s say we have been asked to change the system to record the popularity of articles – we decide that the easiest way is to change our ArticleService to use another dependency to record the request of read of the article.

Obviously as soon as we add another dependency the test code will need to be change too. So what are our choices – 1. give up 2. Put up with it and re-factor all of the tests 3. try and find a way to focus our use of mocked dependencies. 1 and 2 have already been tried! So lets try 3!

How about we automatically create Moq’s for the service – like when we use a DI container, just injecting Moq’s that we can control. Most test/mocking frameworks already have a capability – they aren’t too tricky to build, but as we are using Moq how about we NuGet install and use AutoMoq.

 1 [Test]
 2 public void returns_read_article()
 3 {
 4     AutoMoqer moqer = new AutoMoqer();
 5     moqer.GetMock<IArticleRepository>().Setup(r => r.Get(It.IsAny<string>())).Returns(new Article());
 6 
 7     var service = moqer.Create<ArticleService>();
 8     
 9     var article = service.Get(Guid.NewGuid().ToString());
10     Assert.IsNotNull(article);
11 } 

Simple changes – we new up an AutoMoqer tell it to use a Moq with the same Setup for our repository then create our service using the Moqer – tests run green. Now we can add our test for the new feature:

 1 [Test]
 2 public void records_article_read()
 3 {
 4     AutoMoqer moqer = new AutoMoqer();
 5     moqer.GetMock<IArticlePopularityManager>().Setup(r => r.RecordRead(It.IsAny<string>())).Verifiable();
 6 
 7     var service = moqer.Create<ArticleService>();
 8     
 9     service.Get(Guid.NewGuid().ToString());
10     moqer.GetMock<IArticlePopularityManager>().Verify();
11 } 

This time we only deal with our new popularity manager (I know tragic!) – obviously the test fails – so we step by step change the implementation until we get green tests and we end up with:

 1 public class ArticleService
 2 {
 3     private readonly IArticleRepository _repository;
 4     private readonly IArticlePopularityManager _articlePopularityManager;
 5 
 6     public ArticleService(IArticleRepository repository, 
 7         IArticlePopularityManager articlePopularityManager)
 8     {
 9         _repository = repository;
10         _articlePopularityManager = articlePopularityManager;
11     }
12     
13     public Article Get(string articleId)
14     {
15         _articlePopularityManager.RecordRead(articleId);
16         return _repository.Get(articleId);
17     }
18 }
19 

Our existing test (apart from our re-factor to auto mock) was untouched and still runs green – so we can now amend dependencies to our hearts content only affecting tests relying on those dependencies. So our developer is happy that he can write tests without this issue slowing him down.

Really simple. Before you start thinking this is still a bit kek with all the duplication ‘n stuff – go and have a look at the AutoMoqTestFixture in the AutoMoq.Helpers namespace, then re-factor again!

It's fair to say I have a love hate relationship with Logging. I love when logging can help identify the issue - when it's enough but not too much! But I hate when logging code obfuscates the reason of the code. Go on just think back to all those code bases you have seen where littered with Log.Write statements that were clearly put in as the result of a very pressured debug session - not that you would have ever done this of course...

One of the simple things I like to do early on in projects is build in a capability to 'configurably' log method entry and exit within my code base. With parameter details this is more often than not enough for those times when your 'actual' logging isn't giving you enough to diagnose the issue. This allows me to essentially ignore logging at the start of development, as I can use this approach to help diagnose, and steer my 'actual' logging to make it targeted and useful rather than overly verbose and redundant!

I have blogged about this approachbefore (apparently quite a while ago) - but as there have been some significant breaking changes to Interception in Unity v 3.5 and the Enterprise Library v 6 logging application block, I thought a refresh was in order.

This approach is obviously possible and probably just as easy with other containers and logging tools - so there really is no excuse for random Log.Write statements everywhere!

So lets start from scratch - first we are going to add the NuGet package Unity.Interception which will also bring in the Unity package. We create a container and register a simple test interface and implementation.

UnityContainer container = new UnityContainer();
container.RegisterType<ICommand, TestCommand>(); 

We resolve that and execute our test command:

ICommand command = container.Resolve<ICommand>();
command.Execute(); 

Simple - so now we have a container we can look to add the configurable logging capability we want.
We are going to use Enterprise Library for logging and also the configurable policy - so add EnterpriseLibrary.Logging and
EnterpriseLibrary.PolicyInjection packages.

Enterprise Libarary logging needs to be configured, for flexibility I tend to use the Xml configuration approach using the config tool that comes with the full binary download, but it can also be done using a fluent interface. There is plenty of documentation on getting started with the logging application block. Because in this simple approach we are going to use the static LogWriter we need to bootstrap logging after we have configured it by calling:

Logger.SetLogWriter(new LogWriterFactory().Create()); 

Then we need to change the creation of the container to enable the Interception extension:

UnityContainer container = new UnityContainer();
container.AddNewExtension<Interception>();
container.LoadConfiguration(); 

Note that we are loading our external configuration LoadConfiguration - which at this point it will fail as we haven't yet sorted the Unity config (be patient).

We also need to change our registrations to add our interception behaviour - we are just using a simple interface interceptor here as our code will be all interface driven!

container.RegisterType<ICommand, TestCommand>(
	new InterceptionBehavior<PolicyInjectionBehavior>(),
         new Interceptor<InterfaceInterceptor>()); 

Obviously this could make registrations a bit verbose - so you can push this it into an extension method.

The configuration is the only fiddly bit. Unity has a fluent config api so we could add:

            container.Configure<Interception>()
              .AddPolicy("logging")
              .AddMatchingRule<AssemblyMatchingRule>(
                new InjectionConstructor(
                  new InjectionParameter("Library")))
              .AddCallHandler<LogCallHandler>(
                new ContainerControlledLifetimeManager(),
                new InjectionConstructor(
                  9001, true, true,
                  "start", "finish", true, false, true, 10, 1)); 

after the container construction. This configuration adds a policy (called "logging") with an assembly matching rule that intercepts everything from assembly "Library" and uses the LogCallHander (from the policy injection block) to log before and after (check out the constructor of LogCallHandler to see what all those bools and ints do!). Useful, but ideally we want to be able to drop the policy in without code change.

So using UNity design time configuration - we create the following Unity config section:

<unity xmlns="http://schemas.microsoft.com/practices/2010/unity">
    <sectionExtension type="Microsoft.Practices.Unity.InterceptionExtension.Configuration.InterceptionConfigurationExtension, Microsoft.Practices.Unity.Interception.Configuration" /> 

    <container>
      <extension type="Interception" />
      <interception>
        <policy name="logging">
          <matchingRule name="library" type="AssemblyMatchingRule">
            <constructor>
              <param name="assemblyName" value="Library" />
            </constructor>
          </matchingRule>
          <callHandler name="LogHandler" type="Microsoft.Practices.EnterpriseLibrary.Logging.PolicyInjection.LogCallHandler,Microsoft.Practices.EnterpriseLibrary.PolicyInjection">
            <constructor>
              <param name="eventId" value="9001" />
              <param name="logBeforeCall" value="true" />
              <param name="logAfterCall" value="true" />
              <param name="beforeMessage" value="Before" />
              <param name="afterMessage" value="After" />
              <param name="includeParameters" value="true" />
              <param name="includeCallStack" value="true" />
              <param name="includeCallTime" value="true" />
              <param name="priority" value="3" />
            </constructor>
          </callHandler>
        </policy>
      </interception>
    </container>
  </unity> 

  don't forget to add the section definition into configSections:

    <section name="unity" type="Microsoft.Practices.Unity.Configuration.UnityConfigurationSection, Microsoft.Practices.Unity.Configuration"/> 

This is doing the same as our fluent configuration - now you can see the named call handler constructor arguments being passed to the LogCallHandler which is applied as part of the "logging" policy with an assembly matching rule. We can then amend the Enterprise Library logging config to allow capture of parameters.

Now we can simply change the config file to remove the logging policy totally, or introduce more specific matchingRules to target the area of the code base that we haven't logged properly yet!

The more businesses and development organisations I work with, the more I see common problem “themes”. In this series of articles I am trying to highlight the problems and show the potential solutions to some of these common symptoms.

“The team can’t deliver anything!”

“They are always late!”

I usually hear this as a general perception from those outside of the development team or those not involved in the day to day delivery of software.

Perception is reality- so there is a problem. What can we do.

Clarify goals and vision

If the development team can list *loads* of stuff they have done, then it may be that the delivery focus is off – so stuff is going out the door, but it’s not what the business need. Ensure the development goals for the deliveries are clear, valuable, understood and agreed by everyone not just the team. Whilst the development team may care that we are “re-factoring to remove the anti-pattern generic repositories to remove the leaky abstraction” (or whatever) – you know that means nothing to the guy selling it! Make sure the goals and vision mean something.

Advertise

It may be that stuff is actually getting out the door, but no one knows. Obviously you need to make sure people know the team have solved this problem or added that feature. This becomes exponentially easier once you have clarified your goals and vision, ensuring they are aligned with business value and understood by all. In fact ultimately if you can aim to measure progress using delivery of goals and value then advertisement is built in!

Do less to do more

When the team are actually really struggling to ship stuff that needs a bit more investigation. One of the reasons I find often is that teams and even organisations trying to do too many things concurrently. If the team can list 50 projects that they are working on, chances are they aren’t going to deliver any of them to time as they are spending most of their time multiplexing. Kanban tells us to limit the amount of work in play to maximise the output. This works in small scale for the team’s day to day work or in larger scale for the organisational release planning. For me this is all about focus, start with one thing - the most important – complete it, then move onto the next. Only attempt to do more stuff in parallel when you have learned how to deliver successfully – even then be mindful not to fall back into the trap of trying to do too much…

Dramatic title I know – this is really a post about lessons learned and trusting your TDD instincts (your TDD spidey sense!).

First of all I need to explain some history. In the distant past I was leading part of a programme for a customer with multiple teams building software iteratively in an agile manner. There were two teams building in our company - as ever we were all trying to do things to the best of our ability and knowledge. During one of the sprint reviews that were part of our development heartbeat I noted that one part of the system had been built in a, shall we say less than optimal way, with unit tests that were, er, fragile! In essence it was one big lump of code (pretty unreadable), with tests that had clearly been engineered after the fact – testing everything from the top level as large scale integration tests, very, very slowly. My TDD spidey senses were definitely tingling!

But. It was a relatively small bit of code, it worked to spec, and we weren’t exactly running perfectly to schedule. Despite knowing it was an impact to us all as the large tests were slowing down our CI - I was persuaded that we take it on as technical debt, and funnily enough the debt was never paid back!

Now returning to the not so distant past - I returned to the customer to do some consultancy work and spoke to them about the code they had inherited. Largely they had got on really well with the code base that was handed over, with one obvious exception! Any guesses? Talking to them, they had needed to make changes, attempted to but struggled, describing the code as unreadable and unmaintainable – with the tests being more hindrance than help. Apparently after spending a couple of sprints trying to re-factor, they decided they needed to rip and replace!

As if I didn’t already have enough reasons to trust my TDD spidey sense…

So perhaps a better title would have been something like “When TDD is done badly it can seriously impact the maintainability of your code base” or “Just saying you do TDD doesn’t mean your code will be any good” – no where near as dramatic though!

Recently (for reasons that seem to ridiculous to document) I needed to see if I could use the 11.2 version of the Oracle client from an NHibernate application without impacting an installation of 11.1 already on the machine. After a little bit of investigation (and honestly some trial and error) discovered that the Oracle client can delivered as a packaged part of a .NET application. In my case this allowed the client to use a later version than installed on the server, but also allows the use of the Oracle client without having any client installed.

Firstly you need to deliver the necessary binaries to the bin directory, so from the Oracle downloadsget the right version of the xcopy install binary for your platform. In this case I was using the 32 bit version ODAC112030Xcopy_32bit. The binaries required are:

oci.dll from instantclient_11_2
Oracle.DataAccess.dll from odp.net4\odp.net\bin\4
oraociei11.dll from instantclient_11_2
OraOps11w.dll from odp.net4\bin

Without the standard TNSNAMES lookup process there are several alternative approaches to connection configuration:
 

Easy connect

The easy connect option allows you to supply enough to connect to the instance; this is limited to server, port and instance not allowing some of the extended features supported by local (TNS) naming.

<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2">
  <session-factory>
    <property name="dialect">NHibernate.Dialect.Oracle10gDialect</property>
    <property name="connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
    <property name="connection.connection_string">Data Source=oracle-server:1521/orcl;Persist Security Info=True;User ID=SCOTT;Password=TIGER</property>
    <property name="connection.driver_class">NHibernate.Driver.OracleDataClientDriver</property>
  </session-factory>
</hibernate-configuration>

Embed the TNS entry in the connection string

You can embed the connection details from you TNSNAMES.ORA directly in the connection string. All you need to do is put the TNS entry directly in data source on a single line.

<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2">
  <session-factory>
    <property name="dialect">NHibernate.Dialect.Oracle10gDialect</property>
    <property name="connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
    <property name="connection.connection_string">Data Source=(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = oracle-server)(PORT = 1521)))(CONNECT_DATA = (SERVICE_NAME = ORCL)));Persist Security Info=True;User ID=SCOTT;Password=TIGER</property>
    <property name="connection.driver_class">NHibernate.Driver.OracleDataClientDriver</property>
  </session-factory>
</hibernate-configuration>

This lets you use any additional TNS stuff (like load balancing), but is obviously pretty unwieldy from a deployment and maintenance perspective!

Local TNSNAME.ORA

You can actually deploy a TNSNAMES.ORA file into the bin directory and have connection details picked up from here. So the configuration will just use a data source with a TNS name value (in this case test).

<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2">
  <session-factory>
    <property name="dialect">NHibernate.Dialect.Oracle10gDialect</property>
    <property name="connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
    <property name="connection.connection_string">Data Source=test;Persist Security Info=True;User ID=SCOTT;Password=TIGER</property>
    <property name="connection.driver_class">NHibernate.Driver.OracleDataClientDriver</property>
  </session-factory>
</hibernate-configuration>

which refers to a value in TNSNAMES.ORA file delivered to the bin directory containing:

TEST =
 (DESCRIPTION = 
   (ADDRESS_LIST =
     (ADDRESS = (PROTOCOL = TCP)(HOST = oracle-server)(PORT = 1521))
   )
 (CONNECT_DATA =
   (SERVICE_NAME = ORCL)
 )
)

I was recently writing an application that required some background processing to be handled by either a windows service or a console app, it was a multithreaded app and I was using Enterprise Library to handle exceptions by policy (logging to event log for monitoring). When I was testing the console app I realised it should really be writing the exceptions to the console as well as the event log – otherwise what’s the point of the console right!

A quick bit of investigation (and I mean really quick) and I find out Enterprise Library logging block allows you to use the ConsoleTraceListener from the System.Diagnostics namespace – so adding:

   1:  <add listenerDataType="Microsoft.Practices.EnterpriseLibrary.Logging.Configuration.SystemDiagnosticsTraceListenerData, Microsoft.Practices.EnterpriseLibrary.Logging, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"

   2:      type="System.Diagnostics.ConsoleTraceListener, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"

   3:      name="System Diagnostics Console Trace Listener" />

to the config of the console app, and configuring to use this listener as well was all that was needed.

During a recent prototype development we found out we needed a decent search solution - normally this is right where we would turn to Lucene.Net. Lucene.Net is great, but does have some code overhead associated to it (managing indexes etc.), so the fact that we were already using MongoDB and they had just introduced a beta feature for text search (as of version 2.4) seemed to good to overlook!

Upgrading to the latest version of MongoDB was simple and totally issue free, and the instructions for enabling the feature and setting up the indexes required in http://docs.mongodb.org/manual/core/text-search/ were very clear. All that was required was to start the process with a parameter to enable the text search, and then creating indexes which for my requirements were simple from the console:

db.Activity.ensureIndex(
                           {
                             Title: "text",
                             Description: "text",
                             AlsoKnownAs: "text",
                             Keywords: "text"
                           },
                           {
                             name: "ActivityFullTextIndex"
                           }
                         )

The above creating a text index for my collection (Activity) on the Title, Description, AlsoKnownAs and Keywords properties.

So after the indexes were created using the console, all that remained was actually using the search feature.

There is no direct implementation (yet) in the official C# driverso it requires calling the command directly. Reading the unit tests shows just how easy this is. The search operation in all its glory (not that much glory to be fair):

   1:  public IEnumerable<T> Search<T>(string search) where T : class, new()

   2:  {

   3:      var textSearchCommand = new CommandDocument

   4:          {

   5:              { "text", typeof(T).Name },

   6:              { "search", search }

   7:          };

   8:      var commandResult = _database.RunCommandAs<TextSearchCommandResult<T>>(textSearchCommand);

   9:   

  10:      return commandResult.Ok ? commandResult.Results.OrderBy(t => t.score).Select(t => t.obj) : null;

  11:  }

The command document is created to search the collection identified by the supplied generic type T and is supplied a search term. Using _database (an instance of MongoDatabase) we run the command using RunCommandAs returning the results in TextSearchCommandResult (coming soon). In this prototype code if the command result is Ok we return the result objects – ordered by score – obviously this is passed on to render the search result.

So you are now thinking that TextSearchCommandResult must be really complicated, ‘cos the search bit was a doddle right:

   1:  public class TextSearchCommandResult<T> : CommandResult

   2:  {

   3:      public IEnumerable<TextSearchResult<T>> Results

   4:      {

   5:          get

   6:          {

   7:              var results = this.Response["results"].AsBsonArray.Select(row => row.AsBsonDocument);

   8:              var resultObjects = results.Select(item => item.AsBsonDocument);

   9:   

  10:              return resultObjects.Select(row => BsonSerializer.Deserialize<TextSearchResult<T>>(row));

  11:          }

  12:      }

  13:  }

  14:   

  15:  public class TextSearchResult<T>

  16:  {

  17:      public T obj { get; set; }

  18:      public double score { get; set; }

  19:  }

Wrong. Using the CommandResult base class the heavy lifting is done. All that is done here is to deserialize the identified objects into a simple TextSearchResult wrapper (simply to include the score for ordering – there is more info returned, but this prototype only needed the score).

Pretty quick to get text search up and running. Clearly this is still in beta, and doesn’t have the depth of the Lucene.Net implementation yet. Definitely one to keep an eye on though.