Over the years our company has had a great deal of success helping customers retire and replace legacy monitoring platforms in favor of cheaper, faster and easier to manage systems. However, you don't simply fall out of bed and replace your Netcool Impact system with NOC HERO™ or SevOne with Zabbix. There is planning and preparation that goes into making this happen. That planning starts with gathering requirements to make sure your stakeholders are getting what they want and the system meets the needs of the business.
You may be thinking, "but we are migrating, it's like for like. We just need to ensure functionality is there in the new system and test it out." Wrong. Nothing could be further from the truth. Requirements gathering for a migration to a new platform is more important than in a greenfield. In a greenfield, expectations are there, but there isn't any basis for comparison. In a migration, your user community has things they like and things they don't like in the current system. However, this has to be balanced against the needs of the business in terms of cost. The only way to ensure expectations are met up and down the chain of command is for the monitoring tools team to have a well organized, well executed requirements gathering strategy.
Unfortunately, I often see poor requirements gathering lead to what is perceived as a "failure" in execution when doing not only a migration but any implementation. Here are the ten mistakes I see most often and how to avoid them:
No Clear Mission or Goals
Without a leader (who has not just responsibility but actual authority) who can set clear, enforceable goals, a migration project is pretty much doomed from the start. A discussion on mission statements and goal setting is outside of the scope of this blog, however, you can find some nice reading on these topics here and here. While goal setting seems obvious, you would be surprised to find out how many times we have seen a team try to migrate a system from one platform to another and had no idea why they were doing it. If you asked three different team members why the project was important, you would get three different answers, and none of them were complete. Without every single contributor understanding every goal of the migration, you will never be able to gather requirements because you won't be able to intelligently talk to your stakeholders about why you are doing what you are doing. Your stakeholders will simply expect the status quo (or better), which may not be the driving force behind the migration.
How to fix this
A good start is for the leader of the project to create goals for the project and publish them in a prominent place. Don't email them, they will get lost. Put them on a butcher block of paper and post the goals in a common area. Make sure they are visible at every meeting and not just for your team, but for any user or stakeholder that may pass by. You should especially do this if the goals are oriented to saving cost at the expense of functionality expected by these same users.
Stakeholders Not Identified (Properly)
Speaking of stakeholders, this leads me to the second major mistake I see during migrations. You can rarely get anyone to agree on who the stakeholders are for a monitoring migration project. Too often I see a project that starts out with several Q and A sessions with a few different groups of current users (NOC, engineering, server team, applications, etc) that result in a nice document that the monitoring tools team calls "requirements". The project progresses and finally ends. A month later, someone from team XYZ complains that the new monitoring system "stinks" because they cannot get the report/data they wanted from the system or someone from application development thinks the system is "weak" because they cannot send their alarms into the system. Invariably these people are wrong, but the perception is still there and the reputation damage is done. Word starts to spread and before you know it, the new system is branded (unfairly) as a "failure".
How to fix this
First, let's be clear on who a stakeholder is as it concerns monitoring. A stakeholder is anyone who needs to send information to or receive information from the system, period. This includes a network engineer that needs to send a trap all the way up to the CEO who needs a dashboard and everyone in between who touches or even looks at the system. This sounds like a lot of people...and it is. It isn't called "cubicle" monitoring....it's called ENTERPRISE monitoring. If your enterprise has a lot of people, you are going to have a lot of stakeholders. We suggest that at the beginning of a migration, someone that is at least at the level of vice president put out a notification to all departments about the impending project and requirement for participation. Next, the project leader should get an organization chart of each group that will be contacted and create a schedule of meetings with enough buffer time to accomodate new stakeholders that they didn't know about. If someone within a group is designated as a "representative", that's fine. However, it needs to be documented who that person is representing. Too often, I see a "representative" speak about requirements on behalf of a group, leave that group, and then find out that this person didn't really "represent" the group as well as they should have.
No Stakeholder Education
Once you have identified your stakeholders, you need to engage them to gather requirements. However, many requirements efforts start in the wrong place. Too often I see teams fail at requirements gathering before the first question is even asked. More often than not, the monitoring team has made the (poor) assumption that not only does the stakeholder understand monitoring, but they also understand why the project is taking place. This leads to the gatherer of the requirements asking questions that are mostly misunderstood and answered incorrectly or not at all by the stakeholder. Time is simply wasted. Worse yet, these inaccurate and incomplete requirements are then folded into the project which are later applied to deliverables which ultimately yield functionality with no value.
How to fix this
Remember those goals and mission statement we talked about earlier? Those should be the first things you bring up to frame the meeting. Never assume that your stakeholder understands the purpose of your project or even what the monitoring does today. A few helpful questions to get started:
"What do you know about what we do today on the monitoring team?"
"Did you hear about our migration project?"
"Do you know why we are migrating off of platform <name>?"
"Do our goals make sense?"
No Requirement Standard
It seems like a silly question, but how do you know if what your stakeholder just told you is a requirement? You are probably answering "well, I asked a question and got an answer", or "they told me this is what they expect from the system". I have heard people describe a requirement as an "ask" or a "piece of functionality" or "something the system needs to do". These are all important aspects of a requirement, but let's look at it a different way. Requirements need to manifest themselves in a future system as functionality. In order to get that functionality, the project leader needs to convert requirements into deliverables that are then assigned to a team.
Too often I have seen teams ask a question, get an answer to the question, and realize later during project planning that the answer simply didn't translate to deliverables. It wasn't clear if the answer was one or multiple requirements. The team also had no idea why this functionality was needed or if it mapped to their original goals for the migration. Some responses didn't have a clear state of completion or criteria for success. All of this led to projects that had unclear or even impossible deliverables that would never yield useful functionality after the migration.
How to fix this
Once you gather (what you think is) a requirement, you should put that requirement to the test. The INVEST mnemonic is usually used with Scrum product backlog items (PBIs). We like using INVEST. to test if something is a requirement or not.
Independent: Are two requirements so tightly coupled that they cannot be moved around? If so, try to combine them.
Negotiable: Is the stakeholder willing to discuss the requirement? If the stakeholder is telling you how something should be done vs. what they want and why, you don't have a requirement.
Valuable: Does the requirement from the stakeholder support your migration initiative and goals? Does it add value to the business and if so, how?
Estimable: What level of effort could you assign to the requirement? If it is unknown, you need to redefine the requirement.
Simple/Small: Can you perform the tasks for this requirement in 30 days or less? If a single requirement's deliverables take longer than 30 days, you probably have more than on requirement.
Testable: How do you know when you have finished meeting the requirement? If you don't know, go back and redefine the requirement.
Poor Questioning Techniques
We have been asking questions our entire lives. Hardly anyone would look at themselves and say "I am bad at asking questions". However, when it comes to gathering requirements, asking questions is a skill that takes years to perfect. I have seen the wrong questions asked, leading to answers that would never translate to a valid requirement. I have seen good questions asked that lead to incomplete answers. Bad questions lead to bad requirements which lead to a bad project which lead to an unsuccessful migration.
How do you know if your questions bad ones or if you are missing the good ones? Here are a few mistakes I have seen people make:
1) Questions are focused on the technology and not stakeholder needs. Technology is important, but your stakeholder doesn't really care how meet their need:
BAD: "What types of traps does your system send?"
BETTER: "What types of activity do you need to know about and when do you need to know?"
2) Questions don't get to the business needs. You must put your stakeholder to task about why they need something. These questions are important because they can determine if these requirements align with the goals and mission we discussed earlier (see "Valuable" above).
3) Questions don't help define an end state. Too many times, I will hear Q and A sessions that go back and forth with really good data exchanged, but no clear answer to the simple question "how do we know when we are done with <insert requirement here>?" To avoid this, make sure that you are being absolute and precise in your questions. Do NOT be afraid to put your stakeholder on the spot! Come right out with it: "So, to be clear here, we will have satisfied your requirement if you get an email with this subject, this body, and this graphic in the body if this happens at this time under these circumstances, is that correct?"
How to fix this
An entire blog entry can be written on this topic (and probably will be in the future :), but for now, if you can avoid these three mistakes, you will go a long way to asking good questions.
No Definition of Done
We just touched on this topic when discussing the problems with questions. Too many times I have seen what appear to be really good requirements with one exception....there is no definition of done. No requirement is complete without an explanation of what it takes to actually finish. Seems like a novice oversight, but this is far more common that you would think and leads to massive scope creep on not just migration projects, but projects of all types.
How to fix this
Write down a bullet list of what it takes to finish the requirement. Make sure your bullets are clear, specific and agreed upon by your stakeholder. If you cannot gain agreement on the definition of done, get your project leader involved to make a decision.
No Effort Estimate
Remember when we talked about "estimable" above? This is another common mistake in requirements gathering. If you have collected a requirement, the first question you need to ask is "how long will this take?" If you don't know how long a requirement will take to implement you don't understand the requirement. Our high watermark is 30 days, with the average requirement being 7 days. If a requirement takes more than 30 days, you probably have more than one requirement.
How to fix this
Play project poker with your team. Project poker is a great way to plan and estimate how long a requirement will take. Once you have all of your requirements estimated, your project leader can easily forecast how long each deliverable of the migration will take.
No Requirement Owner
In this case, ownership does not mean the person who is assigned the deliverables related to a requirement. Organizations rarely have an issue with assigning work. The issues arise around coordination and communication around that effort. This is why a requirement needs an owner. The owner of a requirement is responsible for a requirement's completion as well as the communication back to the project leader about any obstacles. Without an owner, requirements will get worked, but blockers and obstacles never get communicated and project delays are inevitable.
How to fix this
Establish an owner for each requirement (not the project leader) and make sure that the protocol for communicating delays, ambiguities, and blockers is clear. When you are migrating systems this is especially important. If the system you are migrating from operated one way and your new system operates a new way, you are going to have a blocker at some point.
Documentation Failures
This is mistake that people wrinkle their brow at when I talk about it at first, but it becomes pretty clear when you peel back the details. How can you mess up documenting requirements? You write them down, distribute them to the team, and so on, right? Pretty simple? Not quite. Here are a few problems I see when people document requirements, especially when it pertains to a migration:
1) Requirements are usually quite fluid in the beginning. You have a meeting, gather requirements and write them down in MS Word, Excel, etc. 2 hours after the meeting, your stakeholder forgets to mention something, sends you an email. An email thread is started. Doesn't seem like a big deal, until you multiply this by 14 stakeholders and 23 email threads over a 2 week period. Before long, you spend all day and night trying to match up time stamps of emails with your 12 MS Word versions. This is a recipe for failure.
2) Requirements for what functionality gets migrated will shift in priority depending on any number of factors (cost, time, management decisions, vendors etc). So now, on top of trying to just maintain accuracy in your Word doc or Excel spreadsheet, you now have to keep track of which requirements you need to do as part of which phase. This will again lead to disaster as you have no idea what is "in" and what is "out" for any given phase.
3) Collaboration and distribution of requirements breaks down completely because of points 1 and 2. Even if you put your document up on a shared site (Google Docs, Sharepoint, etc), you still have so many changes and versions that ultimately conflict with emails and conversations that no one trusts what they see.
4) "Flat" documents such as Excel, Word and so on are not ideal for tracking the life cycle of an individual requirement. When you save a document, it saves and tracks the changes for the entire document, not for a specific requirement. Your 14 stakeholders don't necessarily care about each others requirements, just their own. They don't want to see changes to everyone's requirements. For a migration project, a stake holder wants to know if their requirements are going to be kept in the new system and if so, in which phase they will be included.
How to fix this
We recommend a program that is dedicated to requirements gathering. We use JIRA extensively for our projects. It allows you to collaborate virtually as you create and build your requirements. This all happens in real time. If a requirement changes, anyone who is affected by the change to the requirement can get an email and respond. There are sections where people can add comments and elaborate on a requirement. Additionally, as a requirement is being worked and turned into a piece of functionality, stake holders can see when they will get their functionality in the new system and know when they can begin using it.
No Adapting for Changing Requirements
Too many organizations treat a migration as a linear process. They see it with a beginning, a middle and an end. Truth be told, no monitoring project should be treated this way, but especially not a migration. A migration to a new platform is inevitably going to lead to new discoveries and learning about capabilities. As such, a migration to a new platform will lead to new requirements. Unfortunately, there isn't a process to revisit old requirements and make sure they are still valid.
How to fix this
The project leader should periodically review the mission and goals. Once the leader confirms that the goals and mission are still on point, the leader should schedule regular follow up meetings to make sure that requirements are not only met, but haven't changed. After the migration is complete, these meetings should take place at least once a week. As the stakeholders get more accustomed to the new platform, these meetings can happen less frequently, but they should still happen at least once a month.
Chris founded and has been the majority owner of MKAdvantage and VirtuOps since 2006. He got his start in communications as a Signal Officer in the U.S. Army. For almost 25 years Chris has built software solutions and provided consulting services for the United States Government and fortune 500 companies all across the globe.