Penetration testing is an exam that cyber security experts tout for finding out the true strengths and weaknesses of an organization’s personal and technology defences.

However, if your organization doesn’t have a mature security program pen testing is a waste of time and money, two veterans warned infosec pros Tuesday at the annual SecTor cyber security conference in Toronto.

It was one of dozens of pieces of advice presenters Tim West, chief risk officer of Saint Louis-based Atredis Partners, and Mark Baseggio, a Toronto-based security consultant gave that CIOs, CISOs and purchasing officers could find handy.

In an interview later West and Baseggio expanded on this and other points.

“If you have a very immature security program and you know it – which most clients do – then that’s a very clear indication you should probably put your money into the building blocks that make you secure rather than a shot in the dark,” said Baseggio. “You already know your network is insecure because you haven’t put any effort into it.”

Hiring a consultant to do a pen test “because it’s a popular thing … you’re just not going to get that much value,” agreed West.

Also, they advise not investing in a pen test if the organization doesn’t have a threat model and understand the threats it regularly faces. That leads to disagreements on whether a particular test is relevant to the company. “If you don’t know what matters to you from a security perspective how are you making decisions to invest in security?” asks West.

“People jump to the gun on scoping the pen test based on ‘You got Web apps? Do you want phishing? How many IPs do you have? What kind of apps were you testing?’ said West.” But the questions should be what are the types of attacks that are relevant to your business. That will lead to which applications to test, which subnets, which users if social engineering is being used.

The biggest mistake organizations make when looking for a pen testing service is “buying based on factors that aren’t necessarily related to value – buying based on brand, size, or other things that in our industry don’t directly deal with buying from an expert,” said West. It’s easy to buy from a large IT or financial accounting firm, he said, but that comes at a cost. “Without really scruitinizing and trying to understand the marketplace its easy to make bad decisions.”

Similarly, they advise not hiring a pen testing firm if you don’t have a threat model.

Look for experts that will try to solve the particular problems the organization has. “Ultimately when you’re buying security consultancies what matters is the people that are going to be on your project,” West said, “which is as important if not more than the name on front of the page.”

“Regardless of size (of the contracting firm), know who is going to do the project and vetting that individual is going to get you the biggest bang for the buck.”

And don’t forget to check the consultant’s references, West and Baseggio add. “It’s bothersome” that few clients do,” said West.

Ideally the consultant will offer three reference customers that are in a similar industry with a similar testing project. “With the weeks and hours that you spend in the procurement process, the referral activity probably takes you two hours of your life at most and it’s the most valuable two hours you can spend.”

And don’t worry about candor, says West. Most CIOs when talking one on one are “wildly open.”

Ask what went well, what went poorly, did they stay on budget, manage expectations well, did they communicate well, did the deliverables meet expectations, how much value did you get, would you spend that amount again.

Most organizations wrongly believe a pen test is a pass/fail exercise, they said – and worry that a “fail” could damage a regulatory compliance audit. The problem is the organization doesn’t have a risk management program or an understanding of risk in general for security, West said, so any “fail” in the report can lead to argument because the organization sees it as a sign of weakness. “In reality everyone has information security risk today,” says West, “it just depends on how you’re managing it.”

The consultant has to make clear there will be vulnerabilities found, agreed Baseggio. “Pen testers don’t always frame vulnerabilities properly,” he admitted. “Sometimes we put high on this and make it seem like it’s the end of the world, but in the context of their business it’s not really that important.”

For more information see the Penetration Testing Standard.