Life, Teams, and Software Engineering

Template Messed Up

2012-01-19T15:21:00.000-05:00

Ya, so my template is messed up. Apparently backing up and restoring your template in Blogger doesn't guarantee that it will restore exactly as you left it. I'll get to it soon enough, along with some actual content since it's been forever and a half since I posted anything.

Quality is Not an Option

2011-08-22T19:52:00.001-04:00

While I'm hardly the first to talk about the "Tradable Quality Hypothesis" hopefully I can reinforce in some of my readers (all 10 of you :) ) that Quality is not an option. You cannot choose to lower quality (or forego quality producing/assurance practices) in an effort to get more features "done" or to deliver work more quickly. At least not for any realistic amount of time. If you do this, you might as well plan the rewrite into the schedule now.

So What if I Do?

Immediately, you will probably feel little consequence from omitting a few test environments or writing up those features without any unit tests. But I promise it will catch up with you. Failing to test (at any level), continuing to use outdated tools (without a transition plan), and knowingly adding functionality of little to no explicit value contribute to decaying codebases where bug counts and cost of change increase, while overall velocity (i.e. the rate of value addition) will decrease. If you skip testing it now, guess what? You'll end up testing it a lot more when those bug reports start rolling in.

Yes, you might get to be a hero; and everyone loves a hero, right? Maybe. But when a situation arises where heroics are necessary (e.g. long nights & weekends) just to hit normal commitments teams should not celebrate. That person has just set the precedent for what your clients and customers will expect of the team from this point forward. Resist the urge to be a hero and be a member of the team. Pull each other up as best you can to pull it in at the end, but be careful not to introduce large variances that can invalidate your velocity for that iteration.

If your team is delivering less and less because you're trying to catch up with bug reports you may never be able to make up that time. This can not only harm your organization's reputation, but your personal or professional reputation as well. Considering that, is it really worth the risk?

How to avoid the hole and dig out

There are many ways to avoid getting into the position of deciding to trade off quality for short-term schedule benefits.

One word: discipline. Be vigilant that everything you contribute is tested in multiple ways (unit, integration, UI, etc.), have as many people as possible review your work and provide input, and make sure those things keep happening. Yes it's hard and can look to the uninitiated as if you're moving more slowly, but when that thing you just wrote inevitably changes next week you'll appreciate putting forth the extra effort.

Next is education. Everyone should understand (though not necessarily intimately) what goes into delivering functionality. Establish your definition of done and make it well known. Hang it on the walls. Recite it at the beginning of each daily standup. I don't care how people remember it, just that they do. Whenever something is untested, it's not really done yet.

If you've found yourself in this unenviable position, the first step is to admit that you have a problem. Seriously. We're all proud of our solutions but sometimes they need cut down and replanted. There's no shame in it, I promise. There's only shame in voluntary insanity (doing the same thing over and over again and expecting different results).

Next, come up with a plan of how to tackle what is most commonly the real problem: technical debt. Technical debt is any sub-par component of your entire solution space. This could be anything from an inflexible test harness, to untestable or scary to change code, to relying on outdated or unsupported software packages. If it causes you pain on or disappointment on the development side, it likely falls in here. Environmental or issues external to the team should be raised to the team's manager (e.g. Scrum Master) for them to deal with outside the team. You need to identify and manage this technical debt before there is any hope of a sustainable pace of quality and value.

Don't try to make it perfect, just make it better. Don't feel bad about it, what worked for a team of 5 and 100000 lines of code may just not scale to a larger team or codebase. Our products grow and so our development support infrastructure must grow around them.

Remember, we're the professionals

It may give the higher ups warm fuzzies to hear that it can all fit neatly into a little box, but no one will like the feeling later. We do them no favors by making commitments we know we can never keep within any reasonable standard of long term success. Clients can and should set constraints on delivery and tell us what they want delivered, but it's our job to define how best to get there. I don't tell other professionals how to do their jobs when I pay for their services, and I would hope they would advise me on the consequences of taking shortcuts, while avoiding them completely. That is after all, why I'm paying them. They know better than I do.

If there are doubts about making commitments, or about quality, don't be afraid to tell your clients (or your teammates) the truth, or at least what you observe. That's what you're paid for, speak up. Ultimately, even if the clients don't say so, I'm sure if you asked whether or not you should test this feature you just gave them they'd be scared you even asked the question. Quality is always a requirement even if our clients don't list it as a deliverable.

C++ C Function Wrappers - Introducing CFunctionWrapperGenerator

2011-06-29T20:24:00.000-04:00

Dependency-injection in C++ is not convenient, especially when you bring "Free" C functions into the mix. There are several approaches you can use to isolate these functions, but the one I've had the most success with is to create C++ classes whose sole purpose is to wrap calls to C functions. This approach works quite well, both in minimizing the impact to the production code, and simplifying unit testing, particularly when paired with the googlemock C++ Mocking framework.

I've been doing this for some time and have started to feel some pain in the maintenance area. Wrapper functions are starting to be duplicated between wrapper modules, and classes depending on wrappers are importing functions that they'll never call. But most of all, building out and maintaining these wrapper classes takes time and is extremely tedious. So being the lazy developer that I am, I wrote C Function Wrapper Generator (CFWG) to do the heavy lifting for me.

Basically it's a collection of Python scripts that generate C++ classes whose sole responsibility are to provide interfaces to C Functions. They provide a transparent API to the C functions so you don't have to change how you're calling them (much), but the main advantage they provide is during unit testing: you now have a seam to control side-effects and return values of C functions that you would otherwise have had to deal with by hand.

The main interface is a YAML configuration file that looks like this:

Functions:
    - name:          CreateFileA
      real_header:   winbase.h
      include_header: windows.h
    - name:          CloseHandle
      real_header:   winbase.h
      include_header: windows.h
    - name:          WriteFile
      real_header:   winbase.h
      include_header: windows.h
    - name:          CreateEnvironmentBlock
      real_header:   userenv.h
      include_header: userenv.h
Aggregators:
    - name: FileExists
      functions:
         - CreateFileA
         - CloseHandle

This basically tells CFWG what functions you want to wrap, where it can find them, and what header should be included in real code that calls it. You can also specify custom aggregators if you want. This allows you to pull in just the functions you need. In the case of shared libraries this will prevent callers from getting linkage they don't actually use.

CFWG generates three files:

ICWrappers.h - Contains an interface for each C function in the form ICFunction (e.g. ICreateFileA). This also contains an IMasterCWrapper (which multiply inherits from the individual ICFunctions) which comes in handy for unit testing (and my examples) and an interface for each aggregator specified in the configuration. You want your production code to depend on these interfaces.
Component/CWrappers.h - Contains the implementation for each interface listed in ICWrappers.h. These actually call the real C function, simply passing the arguments down. You pass an instance of one of these to any production code that needs the function (or functions in the case of an aggregator).
Mock/CWrappers.h - Contains GMock implementations for IMasterCWrapper in ICWrappers.h and one for each Aggregate interface specified in the configuration file.

/* Unit class (has dependencies) */
class Unit::Foo
{
public:
    Foo(ICreateFileA &createFileA, IWriteFile &writeFile, ICloseHandle &closeHandle)
        : m_createFileA(createFileA), m_writeFile(writeFile), m_closeHandle(closeHandle)
    {}
    void bar();
private:
    ICreateFileA &m_createFileA;
    IWriteFile &m_writeFile;
    ICloseHandle &m_closeHandle;
};
void
Unit::Foo::Bar()
{
    HANDLE handle = m_createFileA.myCreateFileA(
        "testFile.txt",
        GENERIC_WRITE,
        0,
        0,
        CREATE_NEW,
        FILE_ATTRIBUTE_NORMAL,
        0);
    if (INVALID_HANDLE_VALUE == handle)
    {
        throw std::exception("CreateFileA failed!");
    }
    const char *toWrite = "yay!!";
    DWORD numBytesWritten = 0;
    if (!m_writeFile.myWriteFile(
        handle,
        toWrite,
        strlen(toWrite),
        &numBytesWritten,
        0) || numBytesWritten != strlen(toWrite))
    {
        m_closeHandle.myCloseHandle(handle);
        throw std::exception("WriteFile failed!");
    }
    m_closeHandle.myCloseHandle(handle);
}
int
main()
{
    Component::MasterCWrapper wrapper;
    Unit::Foo foo(wrapper, wrapper, wrapper);
    foo.Bar();
}

Now you can write unit tests for Foo and easily test all these conditions. Well, you could before too, now you just don't have to maintain all those boilerplate wrappers anymore.

Check it out on Github and let me know what you think!

P.S. This depends heavily on the cppclean C++ AST generator, so props to those guys for providing a great tool.

Why "User Stories" Failed Us

2011-02-02T18:18:00.000-05:00

NOTE: Please read this entire post, I don't aspire to become the Winston Royce of User Stories. :)

About 7 or 8 months ago our teams decided to try using User Stories as our primary mechanism for capturing requirements. Since then, we have taken our lessons learned and have moved away from them. The reason for their "failure" is laughable really: people couldn't get past the name. There were team members and customers that just couldn't get beyond the fact that they're called "User Stories". People would make cracks about Epics and ask whether it's more like Odyssey or Gilgamesh. In the beginning this was funny, and we had a laugh, but some people just couldn't get past it, and we have since decided to phase out the use of that term.

The levels of success with User Stories in our group vary greatly, but as with any two projects you can't really compare them based on a single part of their process. The team I am supporting has delivered value consistently since the project started and we have done so not because we used User Stories (we did, and to great effect I might add) but because we were (and are) disciplined. Other teams call functionality "done" without having high level tests against it that can be easily repeated. I blame myself for those failings if for no other reason than I should have noticed the signs. Someone has to, right?

On my current team when one of us starts to get lazy the other half of the pair straightens them out and makes sure that everything they thought to test has been tested, and that it's in the appropriate place for our automation infrastructure to get at it. We commit a change and within an hour we've validated (yes, that's validated, not just verified) that latest version against all our user stories to date, against all supported configurations, including protecting ourselves from regression. We are able to answer the question "when will version X be done?" using real-time data because we keep Jira up to date and because we understand what "done" means. We don't fool ourselves into thinking we can do more work in 3 weeks than we've historically proven we can do, and we don't let management pressure us into committing as such. We've learned to how to adjust our sprint caps for personnel fluctuation as team members are temporarily stripped away to work on other things, which happens quite often.

Like I said, it's not the tool's fault. It's unfortunate that something so trivial could kill its use in our process. Now we work on "Features"...that contain shorts blurbs about something succinct that the system should do...and are estimated using points... Sound familiar? User Stories are no more or less valuable as an artifact than "shall" statements in more traditional requirements management methodologies, but they're tight, to the point, and no one fools themselves into thinking they can craft the perfect sequence of words to build the perfect statement. No one even tries. Instead, the focus becomes the user's intent, not the words in the requirements document. And intent is best captured with higher bandwidth forms of communication, like getting everyone in the same room and talking it out. That, I've learned, is the true power of User Stories; even if they're called Features.

AAA vs BDD Structuring in Unit Tests

2010-09-30T18:24:00.024-04:00

UPDATE: I updated this to include updates to the BDD example's test function name. I'm starting to dislike having the called interface in the test name. It's inflexible and unnecessary and ultimately doesn't help the reader all that much.

It's always good when there are people on your team whom you can both learn from and teach things to. Such is the case with my current team. A couple team members have never done unit testing as it's known in the industry today. Mostly just "pound at the interface until I'm comfortable and throw it over to testing to deal with" unit testing.

During our first code review there were a lot of issues with unit tests. I actually prefer to let it pan out this way; this way people actually get in there and try it out, to later see what works, what didn't, and (hopefully) get some ideas from the rest of the team about how to improve it. One recurring theme was that there was a lot of redundant code throughout test suites. Setup and teardown were outright missing! That's good, because now they've seen the problem, and part of the solution is to use those. Another thing was how the test cases themselves were structured. I've come across two widely accepted ways of structuring unit tests:

Arrange - Act - Assert
Given - When - Then

I've personally used both in the same codebase when it makes sense, but I'm wondering if there's more to it than just semantics and readability. With AAA you are more likely to interact with the class under test directly inside your test function:

void interface_context_somethingHappens()
{
    //arrange
    mock1->setCallShouldSucceed(false);
    mock2->addFakeValue("Value");

    //act
    out->interface();

    //assert
    CPPUNIT_ASSERT(somethingHappened);
}

However, I've started to notice that if you read your tests, I mean really read them, then using the Behavior-Driven-Development (BDD) Given-When-Then structuring will actually nudge you towards factoring the real test preparation and calls to the class under test out of your test case:

void SomethingShouldHappenInSomeContext()
{
    givenSomeContext();

    whenActionPerformed();

    thenSomethingShouldHappen();
}

void givenSomeContext()
{
    //configure context
    mock1->setCallShouldSucceed(false);
    mock2->addFakeValue("Value");
}

void whenActionPerformed()
{
    //execute action
    out->interface();
}

void thenSomethingShouldHappen()
{
    //check that what should happen happened
    CPPUNIT_ASSERT(didSomething);
}

Yes, it's more code, and yes, it may just be semantics, but I see something more. The naming alone has suggested that maybe I should remove the details from the test. Not only does this produce well-factored code but, but it pulls communication with the class under test to the boundary of my test suite and away from my test cases. Now, if the usage of this class changes for some reason, I only have to update it in one or two places in my test suite, rather than in every single test case. Obviously this approach may not be suited for every single situation, but like I said, I use both where it feels right. Granted, this is considered a good practice when using AAA as well, but you've got to name those newly extracted functions something don't you?

I'd love to get some feedback on this. What convention does your team follow? Are my observations valid or can it be chalked up to something else?

Autotest for Compiled Languages (C#, C++) using Watchr

2010-09-25T18:47:00.004-04:00

When I was learning Rails I set up Autotest on Ubuntu with Growl notifications, which I thought was a pretty slick idea. On Ruby this whole technique is super easy and efficient because Ruby is an interpreted language; there's no compile time to slow you down, and no other files to pollute your directory tree. Compiled languages don't have that advantage, but I think we deserve some continuous feedback too. Here I'll describe how to configure Watchr, a generic Autotest utility, to run compiled tests whenever a source file in the path is updated. This tutorial will use a C# example, but it's trivial to have it trigger on different file types.

Getting Started

First, we'll need to install Ruby and Watchr. Because I'm using Windows I just downloaded RubyInstaller. Make sure you put the Ruby/bin directory in your PATH.

Next, download Watchr from Github, extract the archive and navigate to that directory. Or you can just download the gem directly, but some people might want to run the tests locally first. The following command will install the gem from the local directory:

C:\mynyml-watchr-17fa9bf\>gem install Watchr

Configuring Watchr

Now that we have all the dependencies installed, we need to configure Watchr. This process is easiest if you already have a single point of entry for your continuous build process, but if you don't it's not that bad and you'll probably want one anyway. Now, at the same level as the directory(ies) containing your source code, create a text file. I usually call this autotest.watchr, but you could call it autotest.unit or autotest.integration if you're into that sort of thing. For now, just put in the following line in:

watch('./.*/(.*)\.cs$') {system "cd build && buildAndRunTests.bat && cd ..\\"} 

Yes, it's that easy. What this is doing is telling Watchr to monitor any files that match the regular expression (in this case a recursive directory search for .cs files) inside the watch() call, and then execute the command on the right. I also have it configured to return to the same directory when it's finished, but I don't know if that's actually necessary. The watch() pattern is what you would modify for different environments. For example, you could use watch('./.*/(.*)\.[h|cpp|hpp|c]$') for a Mixed C/C++ system, or watch('./.*/(.*)\.[cs|vb|cpp|h]$') for a .NET project with components built in different languages. An important thing to note is the $ at the end of the regex. Because it's likely that there will be a lot of intermediary files generated during the build process, we don't want a file which happens to match this pattern that's generated at build time to trigger an infinite loop of build & test (like happened to me). The heavy lifting is done here, but the stuff specific to your project happens in build/buildAndRunTests.bat. Let's take a look at that:

pushd ..\ 
echo Building tests 
"C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.com" Tests.Unit\Tests.Unit.csproj /rebuild Release 
popd 

pushd ..\Tests.Unit\bin\Release
echo Running tests through nunit-console 
nunit-console.exe Tests.Unit.dll /run=Tests.Unit 
popd 

You'll obviously want to customize this to the specifics of your project, but right now it's hard-coded to call Visual Studio 2008's devenv.com (on a 64-bit OS) and build a project called Tests.Unit. For brevity it also assumes that nunit-console.exe is available on the PATH. Not terribly interesting, but that's the rest of the work.

Now to have all the magic happen. Run the following command in a new console window from your project directory:

C:\Projects\MyProject>Watchr autotest.watchr

That's it! Watchr is now monitoring for changes to files that match your pattern. Simply modify any file matching the pattern and watch the whole process set off. Once it finishes, you can hopefully see the results and it will wait for the next change.

Now there's one less thing you have to do during your heavy refactoring sessions, or just with day-to-day development.

SA_OAuthTwitterEngine Search Error 400

2010-09-18T00:47:00.020-04:00

It may be deliberate that this function doesn't work but that doesn't make much sense. I'm trying to use SA_OAuthTwitterEngine as the sole Twitter engine in my app but I'm having problems with the getSearchResultsForQuery: method (yes, I've hooked up the YAJL library). Every single time I send a request using the getSearchResultsForQuery: selector on my SA_OAuthTwitterEngine I'm get the following error:

"The operation couldn’t be completed. (HTTP error 400.)"

MGTwitterEngine produces no such error. You wouldn't think calling this selector would be a problem since it inherits directly from MGTwitterClient and therefore provides the interface, but actually it is. The problem lies in SA_OAuthTwitterEngine's implementation of

- (NSString *)_sendRequestWithMethod:(NSString *)method 
                                path:(NSString *)path 
                     queryParameters:(NSDictionary *)params 
                                body:(NSString *)body 
                         requestType:(MGTwitterRequestType)requestType 
                        responseType:(MGTwitterResponseType)responseType; 

which overrides the default in the base MGTwitterEngine. It's described quite nicely in a comment but does nothing to explain the consequences of doing it this way.

// ------------------------------------------------------------------------------- 
// mdificatfrom the base clase 
// the base class appends parameters here 
// ------------------------------------------------------------------------------- 
//    if (params) { 
//        fullPath = [self _queryStringWithBase:fullPath parameters:params prefixed:YES]; 
//    } 
// ------------------------------------------------------------------------------- 

The problem is that it doesn't consider which type of MGTwitterRequestType is being used before making the decision to construct an OAuth request. Certain actions simply don't require an OAuth-formatted request, and in this case it just plain doesn't work. MGTwitterSearchRequest and MGTwitterSearchCurrentTrendsRequest are two types that I don't believe require authentication, and therefore have no use for OAuth requests. To fix the problem for this particular request type add the following code to the beginning of SA_OAuthTwitterEngine's implementation of _sendRequestWithMethod:

//Defer work to super because these requests don't support OAuth 
 if (requestType == MGTwitterSearchRequest || requestType == MGTwitterSearchCurrentTrendsRequest) { 
      return [super _sendRequestWithMethod:method path:path queryParameters<:params body:body requestType:requestType responseType:responseType];
  }

This certainly beats lugging around an instance of each class, one for reading and one for writing. Hopefully this helps someone out, it was driving me nuts.

After using GeSHi I have now decided that I need a new blogger theme....but one thing at a time. Back to XCode.

OCMock iPhone 4.1 SDK

2010-09-12T21:10:00.002-04:00

I imagine that this is a common occurrence when a new version of the iPhone SDK comes out but I figured I'd put it out there for the droves of development teams who will take the plunge with the update this coming week.

If your iOS project relies on OCMock and you just upgraded to the iOS 4.1 SDK (and you upped the Base SDK on the target to 4.1) then chances are your unit test projects aren't building. To get it working again:

Checkout the OCMock trunk: http://svn.mulle-kybernetik.com/OCMock/trunk
Open the OCMock project in XCode
Change the Base SDK on the OCMockPhoneSim target to iOS Simulator 4.1 (see Base SDK vs. Active SDK for why).
Build the target.
Copy libOCMock.a and Headers from the build/$(CONFIG)-iphonesimulator folder to wherever you were keeping them before.

That should allow you to link and run but if your experience is anything like mine you'll be getting a bunch of unexplained errors like "An internal error occurred when handling command output: -[XCBuildLogCommandInvocationSection testName]: unrecognized selector sent to instance". Highly annoying but ultimately it will still report "X out of X tests passed" although with all those errors I'm less inclined to trust it.

Thanks to Vincent Daubry for pointing me in the right direction when he experienced the problem during the last major update.

Agile Design: Ask, is it still valuable?

2010-09-08T17:18:00.000-04:00

Today I got some feedback from my lead on an early draft HLD document. Being new to the team, I asked whether or not they intended to keep the overall UML static class diagram which there was a placeholder for. He said that he wanted to keep it. I agreed because it could be a good first exposure for someone trying to ramp up on how the system works. What we disagreed on was how and when that diagram is created.

I explained that I have no issue with the diagram as long as it is generated from the code, as opposed to it being created before any code is written. He responded with "up-front design is a software engineering best practice". Right away I could see this was a philosophical difference. I responded that while I agree in principle, I have never seen an end system look like the UML diagram that came before it. There's nothing wrong with that, the team just learns more about the system as it's implemented and takes action to improve the design. They let it evolve. We can spend all day diagramming, but I'd rather not spend time documenting and diagramming details that have a high probability of not reflecting reality when it's all said and done. It just makes the document misleading, and what's more, we'll end up having to reverse-engineer the diagrams from source to sync them up anyway.

In his defense, I'm sure he's not really talking about the same big design up front that agilists abhor, but I still believe that up-front design all the way down to the static class diagram level is silly. If it's a thought-exercise, fine, it's usually helpful to at least identify your high level components in order to develop your metaphor, but don't begin the process with the expectation that it will reflect the final system. You'll be disappointed.

Is all design analysis wasteful? Heck. No. My stance would have been different if there were a lot of unknowns with technology or various functional behavior. The additional analysis can be used to further mitigate the risk of a costly failed implementation. Most of all, it allows the team to learn more before diving in. However, even in this scenario I would recommended a different approach, such as Spiking the feature or developing some examples to drive TDD. I prefer an approach that provides real feedback that I'm on the right track. Diagrams aren't very good at this.

If the team is learning more about the system, that's the value in performing some up front design, but only when the time isn't spent on something they would have easily discovered on their own if they had just gone ahead. When faced with a choice of more up-front design analysis, ask yourself, "Is this still valuable?" If your team will gain something from it then go for it without remorse (there's nothing un-agile about being responsible), but once mostly everyone is comfortable, move on and get to the real (and fun) work.

Processes That Work For You

2010-04-05T20:36:00.000-04:00

All processes have their pluses and minuses. Many teams on each side of the 'agile boundary' think they've got it right, and they may, but it's not which side of the boundary they're on that matters. What matters is that they've managed to find processes that work in context.

Maybe you're on one of these teams. Maybe they're mythical to you. Either way, what a stripped down process framework like Scrum gets you is an opportunity to find what works for your team not just because your processes/practices haven't failed you yet, but because you've felt the pain the absence of that process/practice causes. I'll say that again because I think it's worth repeating. We want to be able to say our processes work not because they haven't yet failed us, but because we know exactly what will happen if we remove them; because we've experienced it first hand. A major advantage of the shortened feedback loops of Scrum (or any iterative process) is that it allows you to frequently look at your problems, find their root cause, and come up with a way to fix it. I've seen time and again that existing "traditional" practices are quite often the remedy. I'm not the first to suggest "home-grown" processes but I do believe it's a good approach, perhaps the best if you can afford the effort and really dedicate to it. Just like we wait to plan and estimate until we have the best information we're going to have (i.e. 'the last responsible moment'), why not apply the same methods to process adoption?

Now I'm not suggesting we start from ground zero, that would be foolish. Obviously we use version control, somehow elicit and track requirements, some form of configuration/requirements management, etc, but those are all just tools. The processes and practices that you choose to employ (i.e. Process Areas) should come from need. Choose a responsible starting point and grow your processes as you find need for them from that point on. If it's required that you meet CMMI Level 3, so be it, but you should still revisit your processes frequently (and hope that whatever governing body levied that requirement is willing to accept the added cost and possible, or inevitable, waste).

It's also perfectly acceptable to discard processes that cost more than the value they add, or to replace them with more appropriate ones. That's the beauty of retrospective. After all, a failure across two weeks or even a month is better than a failure after a year of churning. That is if you can course-correct as necessary.

No matter what side of the agile boundary your team works on, you should regularly evaluate your practices and processes. Ask these questions with your team:

Do we really need to be doing this? What happens if we omit this? Would it be responsible to omit this?
Can we be doing this better? If so, is there anything specific we can implement immediately? Anything we need to investigate further?
Does it feel like we're missing something? Is there a practice or process we can introduce to fill the void?
Is something consistently going wrong? Why x 5

I've found that keeping a consistent eye on these things allows us to make changes quickly and effectively without sacrificing overall quality, or any number of other benefits various processes can provide. All processes aren't all right for all situations. Build what works from experience, from pain, from failure, not just because some book or group tells you so.

Giving Back to the Professional Community

2010-03-21T16:09:00.003-04:00

Inspiration

My brain has had a month or so to think about Jean Tabaka's article, Countdown to Agility: 10 Characteristics of an Agile Organization, in the November/December 2009 issue of Better Software Magazine (I only read it last month). This, combined with my recent exposure to talk of Coding Dojos, Code Retreats, and Weekend Testing in the TwitterVerse have inspired me to get my thoughts connecting these two down in writing. Specifically I'd like to talk about #7 in the article, "Contributing to the Community and Maintaining a Profitable Company".

A recurring theme in the article is the marriage of seemingly competing interests, Work/Life Balance and Constant Delivery, Servant and Leader, Sustainable and Successful, the list goes on. It's a good article, give it a read and see how your organization stacks up. At first glance, giving back to the community seems like it will cost you while doing nothing more than generating good PR. The article describes three communities to contribute to: the local community, the global community, and the professional community.

When you hear "giving back to the community" usually you think of community service. Usually this happens locally and there are a lot of companies that do a fantastic job of this so I won't harp on it. I personally wouldn't know where to start giving back to the global community. Perhaps contributing to relief efforts in areas like Haiti, Chile, and Japan given the recent earthquakes? But something I don't hear a lot about is how organizations give back to the professional community.

Types of Events

I'm going to focus mainly on three types of events here; Code Retreats, Coding Dojos, and Weekend Testing, but anything that carries the spirit and intent of these events I feel should be handled the same by an organization. I describe these as:

A deliberate, purposeful (i.e. goal-oriented) gathering of like-minded professionals coming together to better themselves, their craft, and each other.

I'm going to make a few observations about these events, interpret it as you will. All data is as of this writing (March 20, 2010). I'm going to put more focus on the US groups because I'm biased:

Coding Dojos - Map available here

Lots of groups in Europe and South America
2 groups in Canada
Only 5 groups in the US

Redmond (only open to Microsoft employees...)
University of Houston - appears to have gone dormant as of 2007
Oklahoma City
Pittsburgh Dojo - Last met Sept 2009
Albany, New York

Just looking at the map, the US is pretty far behind with this.

Code Retreats

Boulder, Colorado
Pittsburgh, Pennsylvania
Philadelphia, Pennsylvania
Detroit, Michigan
Floyd, Virginia

These are just the ones listed on their homepage as coming up. Noticeably Pittsburgh is on both lists and noticeably this list is all US. That's pretty great and hopefully the list keeps growing.

Weekend Testing

4 chapters since inception in August 2009 in

Chennai, India
Hyderabad, India
Europe
Mumbai, India

The US is completely missing from this list. I know we have talented testers that would love an opportunity like this to share their knowledge and expand their horizons. I don't know what it would take to get chapters started throughout the US, but given the structure the group has taken on, local or regional technology councils seem like they would be a good place to start.

As an Officer, why should I care?

By "officer" I mean someone holding a position of power and decision-making in an organization. So why should you care? Someone is bound to start one of these up in my area sometime, why should it be us? Here are some big reasons that come to mind:

Your employees will get better (on their own time no less!)
People attending from outside see that you are supportive of your employees and their goals
Networks and ideas will develop between the people attending. And we all know what comes from ideas.
Attendees will be more likely to consider your organization when they look for their next opportunity.
and perhaps more altruistically, you'll be contributing to the advancement of the craft. Something everyone can always benefit from.

So instead of "why should it?", "why shouldn't it?". It just means that when that next big advancement reveals itself you and yours will be ahead of the curve.

Suggestions for Implementation

So hopefully I've piqued your interest. Now you ask, "how can I make it easier for everyone to get off the blocks and get one of these started?" I'm glad you asked. Here are some suggestions:

Do NOT

Force your employees to host one or more of these events.

It may seem like a great recruiting and marketing opportunity, and it can be if approached correctly, but don't make it your primary goal. Pursue it with this as your end goal and it's doomed from the onset. These movements should be grassroots, initiated by people with a genuine desire to implement continuous learning and just get better. You get better with practice, which is what these groups provide; an environment for software professionals to practice.

Encourage people who wish to host events like these.
Remove barriers. Make it easy for people to reserve spaces and market the events in your organization's name. Some lunch money wouldn't hurt either :).
Provide support and resources.

Do these things and you're sure to see the long-term benefits I've described above.

What about your company? Does it support efforts like this? Do you have any additional suggestions for employees wanting to start one of these up?

Is tooling only for the youth?

2010-03-04T20:45:00.001-05:00

Disclaimer: This post maybe have no basis in reality at all outside my team, it's just a question. This post is done with information on a case study of 1.

I moved to the wide world of ANSI C++ after working in C# with the love of ReSharper for the better part of 2 years. I managed but I definitely felt the absence of the great refactoring tool that took the things that should be easy and trivial and made them so. C++ is not that way, especially with old IDEs. So I started searching this past weekend for a C++ refactoring tool that actually supported my IDE. I found one that claimed to support it, Visual Assist X, so I installed it and started playing around with it. It works pretty well for what I need it to do but I haven't fully explored it yet.

Either way, this post isn't about Visual Assist X. Today after I set up my keyboard shortcuts (to match ReSharper, no less) I called my team's senior developer over and showed him what it could do.

Him: Another tool?!? *shakes head* I don't trust tools to do much of anything. They're just more systems with their own bugs.

Me: Yes but what they do they do well. Why not take it for what it is and accept that nothing is perfect?

Notice these are not direct quotes. I remember better what he said than my own response (mostly because I was surprised by his reaction), but that was the general nature of the conversation. He then proceeded to call me "Mr. Tool" and was quick to dismiss it. I was a bit confused by this, but didn't think much of it at the time. I had work to do so I went on my way.

Now, sitting in my living room catching up on back episodes of Legend of the Seeker something creeps up from the back of my mind. Do I really rely on that many tools? Let's list them out:

Development:

Visual Studio
CppUnit (is that really a "tool"?)
Rational PureCoverage for capturing code coverage
Hudson for Continuous Integration
CppDepend, CCCC, CppCheck, SourceMonitor for various static analysis (some do things better/more simply than others)
And now Visual Assist X for refactoring support.

Process:

Jira + GreenHopper
Confluence
Crucible
Fisheye

I don't think this list is unreasonable at all. OK, so maybe the static analysis tools are a bit excessive but I like data, especially when it costs me next to nothing to get it through Continuous Integration.

Let's look back to September 2008 when I joined the team. They were using exactly ONE of these tools, Visual Studio. No unit testing, no continuous integration, no process support tools, certainly no automated testing, and worst of all no feedback mechanisms of any kind until a project ended and you handed it over to the customer to say "I hope this is what you wanted".

Flash to today. We have 20+ configurations in Hudson, our latest project has 90%+ unit test coverage at all times, our system testing is as automated as it can be so our test team isn't overwhelmed, all our documentation is maintained in Confluence, and all our issues and tasks are tracked in Jira.

I feel like each of the tools listed above plays an integral part in my day-to-day work as a developer. Obviously as developers we spend less time in the management systems and use very limited features of them, but does that make them any less important? No. If management can just jump out to Jira to check our status or out to Confluence to answer their question, that's one less thing they had to bother me or my team about. It makes me happy and I don't even know it, and I'm sure they appreciate it too.

Now I finally get to the question. Is the reason for his reaction a generational thing or am I completely off base? Are the youth more likely to find a tool-based solution to a pain point while the more seasoned have just learned to deal with it?

Then there's the other possibility, am I over-reliant on tools? Maybe I could simplify, but do I understand how they work, their purpose? Absolutely. I know what the scope of their functionality is, how they do what they do, what they're NOT meant to do, and how to bend them to my will within the constraints of the tool. Each one of them adds value and there are only 3 of them that any developer on the team really has to know or use; Visual Studio, CppUnit, and PureCoverage...I take that back, PureCoverage isn't necessary for them to understand, they just need to have it installed since every time they run a build it runs the unit tests with coverage. They could completely ignore the results and I wouldn't know the difference.

What do you think? I'm sure there are exceptions as there are with any rule, but are the youth more likely to blaze new trails?

From Waterfall to Agile: How we got there

2010-02-10T19:25:00.012-05:00

Upon Arrival

When I arrived on my contract in September/October 2008 things were much different than today.

Projects took on average 3-6 months to complete, with few checkpoints in between. Stakeholders couldn't get any value out of the systems until the end of the project even if they wanted to.
Projects were developed by a single person with very little, if any, collaboration.
The test group was far disconnected from the development team and was not involved with the project until it was thrown over the fence for testing. The testers also had little to no technical background. This isn't necessarily a bad thing, but for our particular group the need to understand the domain and the systems is important.
Requirements (and the rest of the documentation) were written by the developers, usually after the system was implemented. The only reason it was documented at all was because it was required by the client.
While we were co-located with our customer, there was little involvement for validation purposes and contact with users was difficult and I didn't see the team press for it.
Developer verification wasn't being done on a regular basis, if at all. No unit testing at all, at least not on my team. Another group was using it minimally but I don't know if it was a core part of their practice.

Immediate Feedback

The first thing I do when I start with a new team is see how they operate. If there are things that aren't being done well, or not at all, I make it a point to find the reasons. This helps to point me in the best direction for fixing existing processes and implementing new things while creating the least friction. The reasons can vary, anything from management pressure, general uncertainty, to god forbid, complacency.

Most of the practices the team had no familiarity with. Things like developer unit testing, continuous integration, testing throughout the lifecycle, etc. But other things, specifically the documentation problems, were chalked up to a "sad truth" that we just had to accept as a cost of doing business. We'll see about that.

The simplest practices to implement were those that I could stand up and use myself and introduce my team to later. Those were Continuous Integration and Developer Unit Testing. I stood up a CruiseControl instance on a spare server and integrated CUnit/CppUnit with our current projects and any new projects moving forward. This took some work, especially stripping the core functionality into its own library so CppUnit could use it, but we got there. There was some cost involved with educating the team on how to write unit tests and to check CI often but it was minimal.

To be clear, Continuous Integration came before I introduced unit testing to the team. I did this because I feel automated unit testing adds the most value when the developers can't just forget about it (intentionally or not). This is particularly important when everyone is still learning. CI acts as a failsafe to make sure the unit tests pass on an independent machine. The CI server doesn't care about your feelings. If something is broken, it will tell you with some emails and a nice red block on the build status page (and Emotional Hudson in Hudson's case).

Testing Throughout the Lifecycle

Having just received my CTFL from the ASTQB the idea of testing throughout the lifecycle was fresh in my mind. It wasn't the first time I've heard of the concept (my Software Engineering Principles course in school touched on it briefly) but for the first time I had something actionable that I could readily apply. We were using the V-Model so the transition from project states was a perfect place to implement it. Using the states on the V-Model diagram below (from Wikipedia) we implemented static analysis on documents after the Requirements and Architecture activities (SRS), and the Detailed Design Activities (HLD), while building Test Plans, Procedures, and Cases concurrently with development efforts. The effect of this was two fold

It immediately increased the quality of our systems and their documentation.
It got the test team closer to the development team and got both those groups closer to the customer.

Growing Pains

All of this change didn't come without its problems. After a couple months of leveraging CruiseControl some of the other groups started to take interest. They wanted in on the party! However, CruiseControl was becoming a pain to maintain. Adding new projects required modifying the XML configuration by hand and then restarting the server. This wasn't the worst of it though. I was the only person who knew how to do this, making me the guru. Max Pool at CodeSqueeze has written about gurus before and they're not ideal, especially when they're you. Anyway, after inquiring with the development community at my firm I was introduced to Hudson and my life has never been the same. Hudson made it dead simple for teams to add their own projects and do whatever they wanted with them, all without having to come to me for help.

Project Opacity

After introducing better engineering practices I now turned my focus to project visibility. The management team had a hard time getting a grasp on where our projects were. That is, our project visibility was very low. We had weekly status meetings that were always to the tune of "what have you done for me this week?". The meetings were often unhelpful to everyone and have only recently been done away with (within the last month or so), probably as a result of the increase in project visibility. At least I'd like to think so :). This led me to start rolling out Atlassian's Jira. It allows management to break down work and see exactly what was being worked on in real-time, all without having to bother the delivery team. This didn't come without purchasing hurtles (it cost more than $0) but we've been using it in production for the last two months.

Documentation Management

Our team has some pretty lofty documentation requirements levied upon us by our stakeholders, at least relative to other agile teams. They take our systems and documentation and perform further third-party evaluations to make sure we meet their standards. We are required to provide them with our ConOps, SRSs, HLDs (sometimes), test plans, test procedures, test reports, and user's guides.

Anyone who has attempted to maintain Office documents in Subversion knows how much of a pain it can be. This got me started on the long path of trying to find a better solution. At first, I tried building documentation inside Trac, our wiki at the time, but I quickly learned that it didn't support exporting all that well. Wiki2PDF kind of worked, but broke anytime someone sneezed. That clearly wasn't going to work so I moved on. Then I attended a talk given by another organization on the Atlassian tool suite where I was introduced to Confluence. I was hesitant to consider changing our wiki system at first since Trac was an integral part of our day-to-day work, but Confluence's build in support for Word, PDF, and HTML export changed my mind. Confluence's built in features combined with the Universal Wiki Converter (and some select SQL update statements) made this transition a breeze allowing me to export everything from Trac and import it into Confluence. Everyone left on Friday using Trac and began using Confluence on Monday.

Since then we have imported all our old Word documentation into Confluence and deleted them from our Subversion repositories (a happy day indeed!). We now build and maintain all our documentation for all our projects inside Confluence where we can get it our for distribution at any time. I like to think of the knowledge inside of Confluence as being the Configuration Items and the consequent document exports to be "compiling" that knowledge.

Release Management

Our release management process used to be defined in a very heavyweight (70+ page) CM Plan document that was way heavier than it needed to be. Now it's documented on a single Confluence page that describes how to

Tag a build using Hudson
Export the documentation from Confluence as PDFs and
Store the Hudson binaries and PDFs in version control as a release tag.

Why does it need to be any more complicated than that?

Implementing Agile

While all paths led here it wasn't absolutely necessary to implement agile. The reason for choosing to go with an agile methodology, Scrum specifically, was bullet #1 at the beginning of this post. Ultimately we're responsible for delivering software systems. In the old process 5 months into a 6 month project we had nothing to speak of, yet we would have thrown 5 months worth of funding at it. More specifically, our clients had. If they have a need to use some features 5 months into the project why shouldn't they be able to? We may have implemented those features in month 1, but we can't assure they work since testing wouldn't have started for another week or two. The risk we had was that they would want to do just this. If they came to us in a situation like this our value is likely to be questioned, and in this economy who can afford that?

Enter Scrum. Now we maintain a Product backlog with high level user stories that are prioritized at the beginning of each Sprint. We commit to a small subset of the requirements at the beginning of each sprint and (ideally) take all the stories to "done" within our sprints (typically 2-3 weeks, we're still finding our sweet spot), performing our release processes at the end of each sprint. For us, done means

Documentation has been updated (ConOp, SRS, HLD, User's Guides, Test Plan/Procedures)
Code has been implemented with high (90%+) unit test code coverage (where reasonable) and has been checked in for Hudson to build.
Test procedures that are candidates for automation have been automated in one way or another.
Code and documentation has been review by the team and stakeholders, as appropriate. We use Crucible for code reviews.

See, documentation and Agile can coexist! You just need to include it in your definition of done and be sure your team includes that work in its estimates.

Conclusion

So to wrap things up, here are my suggestions to anyone wanting to implement agile, especially Scrum since it doesn't prescribe engineering practices. Many of these probably hold true for implementing change in general.

Implement engineering practices first. You'll see the fastest impact here, and you'll need these come time to implement Scrum.
Lead by example. Start using the new tools/practices on your own and your team will hopefully catch on.
Align project teams (development, QA) and stakeholders prior to implementing Scrum or you'll have some problems.
If you can, ease your team into it. It's all the better if they can arrive at the same conclusions as you on their own as to why the change is needed and how your fix will make things better.
Don't force change if it can be avoided. Provide a good reason for why this is being done. My predecessor meant well but they forced many heavyweight processes on teams that had no say or warning, leading to processes that were not used. If teams discover its use on their own, or are at least kept in the loop, they're more likely to accept and continue using them.

Do you have any additional suggestions on rolling out change on your teams? Please comment below!

Revisited: Reducing code duplication in rhinomocks

2010-01-11T22:23:00.000-05:00

I've been wanting to come back to this for some time now but, quite frankly, I'm a bit ashamed to even look at it again. I'm sure that by now I'm far enough away from it to be able to observe my work as if it were someone else's, and that's exactly how I'm going to structure this post.

About a year and a half ago I wrote a post titled Reducing Code Duplication in RhinoMocks Tests that now makes me want to scratch my eyeballs out. Take a look, it will make you feel better about yourself :). I suppose there would be nothing if not for progress so I'm revisiting it to go over what I (and most of you I'm sure) see as being entirely wrong with the approach described in that post, and talk about what I've learned since then about mocking and unit testing.

The approach described in that post has several glaring issues; smells if you will:

Branching unit tests, which is a general no-no
The tests are over-specified. The close relationship between test code and the code under test greatly reduces the value of the tests and will make the tests very fragile
The mocks are over-relied upon and have become a Maslow's Hammer. I was trying so hard to isolate the presenter class that I missed the point. I was shooting for metrics while ignoring the business logic the MVP implementation is supposed to realize. I should have been focusing on the interactions between the classes, and should have only isolated to directly test specific, complex logic. Only if the presentation logic was sufficiently complex should the model have been mocked away (and in such a case may not even belong in the presenter :)).
This code isn't really testing the classes in a way that is as close as possible to the way they'll be integrated in production, and as a result are very likely to miss things.

So what have I been doing about it? I've been away from .NET development for the better part of a year working in C++ which I think has been really good for me. C++ has the bare-essential constructs necessary to perform object-oriented design and as a result some of the more modern frameworks are less available. I've only been able to find a single mocking framework for C++, googlemock, but I haven't used it yet. Not because I don't want to, but because it hasn't been necessary.

Since reading Steve Freeman and Nat Pryce's Growing Object-Oriented Software Guided by Tests I've been approaching my unit testing differently. I've started treating my unit tests as an executable specification that is defined before the "green" code is ever written, and now I can't do it any other way. Yes, it's just TDD, but Freeman and Pryce's explanation felt more natural to me. Anyway, their approach calls for acceptance-level tests to be written followed by smaller, more specific, unit tests that verify the details of your specification. So how does this relate to mocking? Simple, following this approach allows my design to naturally evolve into something where mocking becomes the final step, not the first, stretching my test fixtures to the point where run-time information is necessary. Just mock out the necessary classes, provide the stimulus, and we're good. Further, it enables me to keep my classes linked together in the way they'll really be used in production and send my stimuli all the way through the system to verify the final effects/output. Obviously you'll want tests in the deeper regions of your classes, but I've gotten more value from my tests implementing them in this way.

Implicit or not...

2009-08-15T20:55:00.003-04:00

I was in the weekly status meeting for one of our projects when I uncovered an integration requirement that wasn't defined anywhere in our requirements documentation. When I asked the product owner why this was the case, he said that it was an implicit requirement, and that all software we develop should have it.

I'm sure we've all been here at one point or another but when it comes down to it, it's not the PO's fault. It's our job when gathering requirements to bring these types of assumptions to the surface. Requirements documentation doesn't care whether a requirement is assumed or hand delivered from the president, it just needs to be there. Getting the stakeholders involved (really involved) in the requirements process is another blog post altogether. Anyway, implicit or not, make sure it's out there for everyone to see.

Software Testing: A Tester's Role, Part 2

2009-04-08T18:46:00.004-04:00

During Implementation
A tester's role during implementation depends on how close the tester wants to be to the actual implementation of the system under test (SUT). If they are meant to be an independent tester, then treat the code as your Kryptonite. Don't go near it! The less you know about how it's implemented the better. Ideally you should know nothing, but depending upon your operating environment that just might not be possible. At the very least, you had better be able to forget about the how when you go to verify that the system does what it's supposed to.

During the implementation phase of the project, the test team can take a built version of the system and exercise it inside its intended operating conditions. Testing should be performed from the perspective of all human actors in the defined use cases. This protects against the team concentrating too hard on one actor's perspective. These "stagings" can be performed weekly, bi-weekly, monthly, or even daily if they so choose. How often depends on how quickly we want/need to receive feedback.

Hopefully the developers are performing test-first development (for all our sakes), but if they aren't the test team will need to be extra vigilant during these stagings because they are effectively performing the unit and unit integration testing.

An added benefit to staging is that by the time the system tests roll around the test team should be well versed in the system. They should know what it's intended to do, how to make it do those things, and what caused the most frequent failures in the past.

A note on communication. Test plans and procedures should be "public domain" on your project. Everyone should be able to access them at any time, specifically the development team. If the developers know how you intend to exercise the system, they can make sure the system works in those cases. Remember that the point isn't to catch the mistakes of the developers to humilitate them. There is no line drawn in the sand, we're on the same team! The point is to catch problems across the board to protect the project and the people working on it.

The sooner we catch these problems, the easier they are to fix.
The easier they are to fix, the cheaper they are.
The cheaper they are, the less overhead we incur.
The less overhead we incur, the more work we can perform with that excess.

Or we could just take that money to the bank. Either way, less money spent can be allocated elsewhere.

Software Testing: A Tester's Role, The Prelude

2009-04-08T17:10:00.019-04:00

I recently gave a presentation to the test group at work that discusses how our project approaches testing. I'm going to be posting the sections from that presentation over the next month or so. It actually facilitated so much conversation that I only got through 2 of the 5 sections in twice the amount of time I initially set aside for the meeting. But that's good, if people are talking about these process changes/improvements, they're more likely to be utilized. Especially when those talking are managers :).

Anyway, over the next month or so I'll post a new section once a week (maybe more often, no promises), covering what I've talked about. Hopefully I can get some good feedback from people here (if anyone even reads my blog!). I split it up into smaller chunks to reduce the size of the individual posts. These posts are to cover:

A Tester's Role, through the entire lifecycle (3 separate posts)
Test Level Entry/Exit Criteria
System Test Procedure Generation
Test Case Input Generation

So here we go.

A Tester's Role
The job of a tester is different at every stage in development, but the overall goal is to promote and facilitate quality. Traditionally, the role of tester was to click the button...after the system had already been implemented. Today, however, that alone just doesn't cut it. To be effective, testers need to be involved in all stages of the project lifecycle (aka. testing throughout the lifecycle).

Different case studies have found that a bug found after release can cost 500 to 1500 times more to fix than if that same bug had been found at requirements. Obviously this implies that the bug was introduced during requirements, but this isn't always the case and is beside the point. What if someone had been there to challenge what everyone assumed that particular requirement meant? What if they asked the tough and often times, obvious questions? Normally this is the job of the requirements analyst(s) on the project, but if you work on a smaller team like mine, there isn't one. In this case, it must become the tester's responsibility.

During Requirements
Requirements lay the foundation for everything the software is to become. If a foundation is weak, what happens to the building placed on it? Right, it either fails to be built at all, quickly crumbles to the ground, or costs tons of money to fix. A tester's goal during requirements is to ask questions to get everyone on the same page. If two developers each make an assumption about a requirement, they could each make a different assumption. Worse yet, what if both of their assumptions are incorrect? Now we have two pieces of the system to fix instead of just one, not to mention that those pieces will likely be incompatible with each other to begin with (Continuous Integration can help with this problem, among other things).

An Example: We have a requirement which states: "The house shall have a front door."

Is this requirement ambiguous? You bet it is! This is an easy one to make an assumption about, but what is the correct solution? Consider the following (yeah, yeah I know the pictures suck):

Each of these solutions is valid by the above requirement, but which is the correct solution? Even this one could be more correct, depending on the customer's intentions:

The point is that based on the above requirement, all these solutions are valid. At this point, the tester can ask his/her self, how would I verify this? Just check if it has a door on the front of the house? But Where?

From a testing perspective, all requirements should be verifiable and quantifiable.

Who is qualified to answer this question? The architect? The homeowner? The contractor? The painter? Chances are that in this case the homeowner would have the final say (they're paying for it aren't they?), but what if they're building the house for someone else? The requirement should be refined by the primary actor/actors in the use case(s) mapped to this requirement. In this case, it's probably safe to say that the homeowner is the best choice (unless they really are building the house for someone else, in which case we would go to them). So, we contact the stakeholder and ask them to be more specific, and we get the following refined requirement.

The house shall have a front door on the first level.

Which is the first level?

The house shall have a front door on the ground floor of the building.

Where on the ground floor?

The house shall have a front door on the ground floor of the building, in between the two front windows.

And we have a winner!

This requirement appeared obvious at first (as so many do), but upon further inspection couldn't have been much worse. As testers, requirements are our most precious artifact. They give us the means to verify everything that comes after it. In the absence of a dedicated requirements team or requirements analyst dedicated to our team, we need to make sure these requirements allow us to do our jobs. Ideally the developers shouldn't have to guess even once during their implementation, but when they inevitably do they should know who to get in contact with to have their questions answered.

Is the Windows 7 Process to Thank for its Beta Success?

2009-01-18T11:26:00.002-05:00

We all know that a beta release does not necessarily speak to the final product, but it doesn't stop people from raising expectations. If recent praise is any indication, the hype surrounding Windows 7 should get it off to a positive start. A recent post over at ars technica talks about how the recent (positive) media hype should have the opposite effect of the media surrounding Vista at about this point in its release cycle.

I, like Emil, am one of the Vista users who haven't had many problems with Vista since adopting it, but what is the root cause of Windows 7's media praise? Is it simply because there was nowhere to go but up, or were there fundamental process changes that increased quality across the board? If you have any interest in software process and haven't read Larry Osterman's post about how Windows 7 handles changes/features, you should do it now.

The question I have is this. Does Microsoft have the new configuration management and Winmain branch merge controls, resulting in fewer bugs actually reaching the beta, to thank for its recent success? Or is it simply the media blowing things out of proportion again as it did back with Vista? Or something else entirely?

I can't speak from personal experience since I haven't used the beta myself (I'm not an MSDN or TechNet member) but I'm interested in getting people's opinions on this. Come to think of it, the fact that I'm calling it "success" without any first hand experience may be exactly the point.

ActiveRecord::StatementInvalid: Mysql::Error: Unknown column

2008-11-23T14:12:00.004-05:00

So I've just recently started teaching myself Ruby On Rails and have had my share of hurdles. Most of them have involved the magical world of ActiveRecord, because that's where I've spent the majority of my time in the rails MVC architecture. Also because the model is the foundation of any solid application. Anyway, to the point. I have encountered this error several times while running my tests when it goes to populate the test database using the fixtures.

ActiveRecord::StatementInvalid: Mysql::Error: Unknown column 'user' in 'field list': INSERT INTO `users_posts` (`user`, `thread`, `body`, `active`) VALUES ('aaron', 'one', 'Some Text', true)

This coming from the fixture users_posts.yml which would read something like:

post1:
  user: aaron
  thread: one
  body: Some Text
  active: true

aaron is an entry in users.yml, and 'one' is an entry in threads.yml. This is a simple example, but the idea is that users and threads can have many posts. You'll notice from the SQL that was printed out, that ActiveRecord was looking for a column called user, when in reality it's user_id. This is something that ActiveRecord does on its own under the covers, so why didn't it work?

ActiveRecord relies on the associations laid out in your models, so if you mess up or omit the associations inside your models, ActiveRecord will get stuck and assume that the data contained in your fixture is literal (i.e. columnName: data). So the users_posts model should look something like this:


class UsersPosts < ActiveRecord::Base
  belongs_to :user
  belongs_to :thread
end

and the Users model can look something like this


class User < ActiveRecord::Base
  has_many :users_posts
end

and the Threads model something like this


class Thread < ActiveRecord::Base
  has_many :users_posts
  has_many :contributing_users, :class_name 'User', :through => :users_posts
end

But even after setting up these models in this fashion the fixture still messes up? Why? The problem goes back to fundamentals, model names should be singular. Because I named my model users_posts, ActiveRecord choked when it tried to use it to get data into the database. After replacing that model with one called user_post, everything works as expected.

It all falls back to that wonderful Rails principle of convention over configuration. If you stick to convention everything will work just fine. Deviate from it (even a little bit) and you'll be chasing ghosts for hours.

An Adventure in Refactoring Part 1

2008-09-02T08:29:00.011-04:00

In this N-Part series I'm going to chronicle the refactoring efforts of my team and I on my current project and pose some questions that fall in the realm of DDD, refactoring, testing, etc. I don't really have a road map for this series but here is a basic idea of what I'm planning on hitting:

Part 1 - Project History & Overall Goals
Part 2 - Finding Focus - Refactoring, Where To Start?
Part 3 - The Big Picture - Utilizing Visualization Tools to Target Refactoring and Find Patterns
Part 4 - A Flood of Information - What is NDepend Trying to Tell Me?

Lately on my project there have been a lot of good things happening. TeamCity has been brought in for Continuous Integration, NUnit has begun to be used for automated regression testing (paired with WatiN for web testing), Sandcastle has been implemented at build time (nightly) to keep up to date project documentation, and I've been trying to sell NDepend as a refactoring/inspection tool the more I learn how to use it, and more importantly, understand what it's trying to tell me.

Background

Before you can really understand where I'm trying to go and what it's going to take to get there you need some background on where this application came from and what its current state is. The project was initiated more than a decade ago by a group of people who knew how to program, but really weren't developers. It was originally written in ColdFusion and was later ported to VB.NET when the need for a newer technology presented itself. This was a great first step and is necessary for any long term project to survive and continue evolving.

The problem was that while the application now existed in a new object oriented language, the scripting mentality came along for the ride. Therefore a solid object model was not developed at the time of the port and over time any architectural changes that were meant to be enforced fell by the wayside when a release drew near. Since software is simply a means to an end within our organization (software is not our core business) best practices are sometimes not in place or, if they are in place they're not the highest priority.

Flash to today and the team is populated with people who know software. Even the PM has a background in software, specifically with this project. I've been fortunate in the short time I've been here to be placed with a group of open-minded people who are willing to see both sides of an argument and accept (and follow) a path to positive change for both process and design. I've heard horror stories about people being set in their ways and resisting change, even when the benefits are obvious.

The story I've heard over and over again is "we're aware of the design problems with that [class], [assembly], [subsystem], etc. but we just haven't found the time to get in and change it." I imagine that this is a problem in the majority of projects out there that ship a product in iterations (and probably some that don't). You promise X at the beginning of an iteration, implement it during the iteration and then ship out at the end with little to no slack time.

This is all well and good if your definition of 'Done' is implemented, tested and refactored because your estimates will have taken these things into account. However, if your estimates don't include testing and refactoring more than likely you will exhaust your estimate and run out of time. This is the first thing that has to change when attempting to implement these changes with a group who has never worked with them before. Their definition of 'Done' needs to change, otherwise we'll never have the time to really test and refactor.

Ok, I'm going to stop right there. In all fairness, we did/do have QA on our team and our iterations end with a two week system testing buffer, but this testing effort was purely at the system acceptance level. There was no developer testing going on at all, so even when a problem was discovered, assuming it wasn't obvious, it would take a little while to track it down. You could even end up with the worst case where you're running in circles because the error you found is the result of another, more serious error. Now, however, we have regression tests for most of the new functionality and for the pages we've managed to refactor, but there's still a long way to go.

Wading through the source code for the system I come across things that can only be described as code stenches. Yes I'm aware of the term code smell, and that's where I'm coming from. They really do bother me that much. Pages are often entities all their own with some even doing data access directly. To make matters worse there are many places that business logic and requirements are implicitly implemented without proper comments or mention of it in the XML documentation, which is also sparse. Data access is very heavily tied to the UI in that many of the DataGrids are bound to a DataSet which is the direct result of a database call.

So where to go from here?

We have started refactoring our pages to MVP (Passive View) with IoC on existing forms to increase testability/reusability and disconnect the UI from the data and business layers. Don't yell at me, I know the goal of IoC is not testability but it was a solid means to an end. This will likely continue to be the bulk of refactoring throughout this iteration but we have already added a DomainModel project where all solid domain objects should eventually end up. The appropriate services then need to be refactored to be encapsulated within these domain objects.

Reducing Code Duplication in RhinoMocks Tests

2008-07-14T09:56:00.015-04:00

UPDATE: This post was later revisited, but it's still worth reading for an example of what mocking shouldn't be :).

I was recently placed on my first project at my new job (I'll post about that later) and they're having me integrate a bunch of great things into their process. These include automated unit testing, continuous integration and refactoring to patterns. Anyway, I've been writing some of my unit tests using isolation with RhinoMocks and came up with a way to achieve full path coverage without duplicating my Expects between test cases.

Below is a fairly standard MVP example. A presenter method is invoked and makes decisions based on information obtained from the view. Say you are testing the following method using isolation and assume the dependencies on view and model are injected via the presenter's constructor:


public class Presenter
{
        IView view;
        IModel model;

        //...constructors, properties

        public void Foo()
        {
                if (view.val1)
                {
                        //...
                        model.F1();
                        //...
                        if (view.val2)
                        {
                                //...
                                model.F2();
                                //...
                        }
                        else
                        {
                                //...
                                model.F3();
                                //...
                        }
                }
                else
                {
                        //...
                        model.F4();
                        //...
                }
        }
}

Now obviously there are 3 possible paths through this method:

val1 = true & val2 = true
val1 = true & val2 = false
val1 = false & val2 = don't care

Now, my approach to this sort of problem in the past had been to duplicate a lot of test code only to change a single case return:


[Test]
public void CanFooVal1TrueVal2True()
{
        Expect.Call(view.val1).Return(true);
        //...some expects
        model.F1();
        LastCall.IgnoreArguments();
        //...some more expects
        Expect.Call(view.val2).Return(true);
        //...some expects
        model.F2();
        LastCall.IgnoreArguments();
        //...some more expects

        mockery.ReplayAll();

        presenter.Foo();
        //Assertions
}

[Test]
public void CanFooVal1TrueVal2False()
{
        Expect.Call(view.val1).Return(true);
        //...some expects
        model.F1();
        LastCall.IgnoreArguments();
        //...some expects
        Expect.Call(view.val2).Return(false);
        //...some expects
        model.F3();
        LastCall.IgnoreArguments();

        mockery.ReplayAll();

        presenter.Foo();
        //Assertions
}

[Test]
public void CanFooVal1False()
{
        Expect.Call(view.val1).Return(false);
        //...some expects
        model.F4();
        LastCall.IgnoreArguments();

        mockery.ReplayAll();

        presenter.Foo();
        //Assertions
}

The approach I've come up with to solve this problem is to have each test case call a single method which uses its parameter list to take a path through the method being tested (granted this is one of the simplest mocking cases):


[Test]
public void CanFooVal1TrueVal2True()
{
        FooPaths(true, true);

        mockery.ReplayAll();

        presenter.Foo();
        //Assertions
}

[Test]
public void CanFooVal1TrueVal2False()
{
        FooPaths(true, false);

        mockery.ReplayAll();

        presenter.Foo();
        //Assertions
}

[Test]
public void CanFooVal1False()
{
        FooPaths(false, true); //keep in mind val2 doesn't matter in this case

        mockery.ReplayAll();

        presenter.Foo();
        //Assertions
}

private void FooPaths(bool val1, bool val2)
{
        Expect.Call(view.val1).Return(val1);
        if (val1)
        {
                //...
                model.F1();
                LastCall.IgnoreArguments();
                Expect.Call(view.val2).Return(val2);
                if (val2)
                {
                        //...
                        model.F2();
                        LastCall.IgnoreArguments();
                }
                else
                {
                        //...
                        model.F3();
                        LastCall.IgnoreArguments();
                }
        }
        else
        {
                //...
                model.F4();
                LastCall.IgnoreArguments();
        }
}

This makes it so that you only need to change Expects and other information relative to the code you are testing in once place (FooPaths) when the code you are testing changes.

Does anyone else out there have any other solutions to this problem?

2007-09-26T16:32:00.001-04:00

Following my meeting with a few companies at the Penn State career fair this past week I realized that it's about time to start experimenting with Visual Studio 2008 Beta 2. So I downloaded Visual Studio Team System 2008 Beta 2 and got to work. Here's what's new in VS 2008 Team System Beta 2, according to Microsoft:

Windows Vista support
- Provides Windows Vista security compliance. (aka, no more work-arounds for UAC)
- IIS still requires admin rights.

Allocation Stacks
- Collects allocation data (from the perf session property page).
- Available in the Allocation view performance report.

Line-level sampling data
- On by default.
- Checks out the new IP and Line view performance reports (and line information in the Module view).

Runtime control
- Starts up automatically on profiler launch.
- Enables and disables collection from the UI.
- Starts with collection disabled.
- Adds custom marks without editing code.

Filtered analysis
- Filters on timestamp, process, thread, and marks.
- Works just like the work item tracking query system.
- /filterfile (saves the filter from the UI if you need the format).

Diff
- Represents a flat diff of a single column with the ability to set a threshold for what shows up.
- Selects two reports in the Performance Explorer and chooses diff from the right-click shortcut menu.
- Chooses new diff report from the tools menu.
- /diff on the command line.

Improved chip counter support
- Provides new friendlier names.
- Configurable xml files.

PDH (Windows) counter support
- Filters on marks based on what the PDH data tells the user.
- Checks out the windows counters page of the perf session properties.
- /wincounter flag.

Compressed report files
- Created from full reports.
- Generates small compressed files that open up very quickly.
- Right-click the report in the UI and choose Save Analyzed.
- /summaryfile on the command line.

Hot Path
- Calltree and allocation view—shows the user the lowest performing path.
- Right-click on a node or use the buttons on the new toolbar.

Copy to HTML
- Copy from the view and paste into email.
- cntrl-c.

Load/Web test integration
- Creates a load or web test.
- Right-click and create a perf session from the test.

Indigo support
- Profiling tools now supports Indigo.

If you've taken a look at the "The Visual Studio Code Analysis Team Blog" recently you'll see that their team has integrated FXCop very well into Visual Studio with a slew of new features for ease of use/access. The most visible thing to me was done after Beta 2 was released, however, with the addition of the Analyze menu item giving you quick access to the features of FXCop.

Another addition in 2008 is the Test menu. It looks like Microsoft is (finally) trying to make it so that you never need to leave Visual Studio to get your work done. For example, in the past when I've run FXCop on my solutions, I found the UI provided by FXCop to be more usable than the Visual Studio integration provided. The same went for testing. I have been using the NUnit testing framework along side ReSharper since I entered the field professionally and it is extremely simple to run my Test Fixtures: Right Click the Solution -> Run Unit Tests and wait. Another feature that will be very helpful is the ability to run coverage right inside Visual Studio, as opposed to running TestDriven.NET's Visual Studio integration. If you're like me and attempting to make a transition from the NUnit testing framework to Microsoft's here are a few attribute correllations that may help:

NUnit	Microsoft
TestFixture	TestClass
Test	TestMethod
SetUp	TestInitialize()
TearDown	TestCleanup()
TestFixtureSetUp	ClassInitialize()
TextFixtureTearDown	ClassCleanup()

Visual Studio Test Classes appear to work the same way as the ones you would create using NUnit, it's just the attributes you use are different. I copied an NUnit test verbatim (aside from the Assert class method names) and ran it using the Visual Studio test runner and it worked fine.

To mention ReSharper again, if you want to try out Visual Studio 2008 Beta 2 but can't live without ReSharper, you can download their nightly build. According to Ben Hall's blog anything later than v.3.0.2 should be compatible with Visual Studio 2008 Beta 2, but don't take my word for it as I'm yet to test it extensively on my own.