|
PROFESSIONAL SECURITY SCANNER
|
Rocket Science, That's Just Our
Hobby!
|
The main criterion for determining
the quality of a security scanner is, of course, the number of
vulnerabilities it can detect. But a scanner should be judged
not by the number its manufacturer claims, but by the number of
vulnerabilities the scanner actually detects. For example, one
important feature is the scanner's ability to identify services
using non-standard ports, without which vulnerabilities known to
the scanner cannot actually be revealed.
A second criterion is the number of
false detections. There is no doubt that it is better to be
over-prepared, but with a high number of false detections,
security specialists or system administrators waste enormous
amounts of time verifying detections and filtering out what they
don't need.
The third criterion is convenience.
Although some may ignore this as a less important factor than
the others, in the end it saves time and energy by minimizing
human error.
In this
survey the following products were tested:
Before testing, each scanner's
vulnerability databases were updated through the Internet with
the most recent versions.
Servers using the following
operating systems were analyzed by the scanners:
- RedHat Linux 7.2 (Enigma)
2.4.18 SMP
- Sun Solaris 7 (SPARC)
- Windows XP Professional
- Windows 2000 Server
- Windows 2000 Server (with
port mappings from FreeBSD 4.7, RedHat Linux 8 and Windows
XP Professional)
- Windows 2000 Professional
(with HoneyPot installed and emulation of FTP, SSH, HTTP,
POP3, NNTP services)
The last server in the list was
specially configured to make analysis more difficult. When
connecting to all services besides HTTP, it responded with a
service banner. Identical responses were sent to all subsequent
queries. HTTP service responded to queries with identical
replies, with a random choice of "200 OK" or "404 Not Found".
Methods of
Comparison
Vulnerability search quality is evaluated in points as follows:
| Vulnerability |
Detected |
Falsely Detected |
| critical |
+3 |
-1.5 |
| average |
+2 |
-1 |
| available information |
+1 |
-0.5 |
For each false detection, 50% of points of
the bonus for correct detection were subtracted based on the
premise that a false response is not critical, but it does cause
delays in the overall process of vulnerability removal.
Product user interface and operating
convenience were assessed as follows:
- Ability to update scanner
and vulnerability databases from Internet
- Built-in scan scheduler
- Availability of different
scan profiles and capability of creating profiles designed
for specific tasks
- Uses pause function in case
of temporary network problems, as well as resume function
starting from the same point (especially important when
scanning large networks)
- Report formats designed for
use by administrators and management
- Remote scanning capability
through a client module connected to a scan server
Final results
Detailed tables with scan results are cited
in APPENDIX A. Total points are as follows:
| |
IS |
LG |
MP |
Ns |
NR |
Rt |
| Total correct detections |
74 |
39 |
133 |
111 |
39 |
89 |
| Penalty for false detections |
-7 |
-1.5 |
-1.5 |
-16 |
-6.5 |
-7.5 |
Grand total
(with penalty subtracted) |
67 |
37.5 |
131.5 |
95 |
32.5 |
81.5 |
These data are easier to interpret
when plotted on a chart:
Table of total functionality
comparison results:
| |
IS |
LG |
MP |
Ns |
NR |
Rt |
| Update |
+ |
+ |
+ |
+ |
+ |
+ |
| Scheduler |
+ |
- |
+ |
- |
- |
- |
| Profiles |
+ |
- |
+ |
+ |
- |
+ |
| Scan Pause |
+ |
- |
+ |
- |
+ |
+ |
| Report Variety |
+ |
+ |
+ |
+ |
- |
+ |
| Client-Server |
+ |
- |
- |
+ |
- |
- |
As you can see, all the scanners scored
almost identically in convenience and most other features, with
the exception of LanGuard and NetRecon.
For speed of operation, the slowest
product was Internet Scanner, followed by. MaxPatrol and Nessus.
In speed, the three leaders were LanGuard, Retina and NetRecon.
Of course, speed in a security scanner is among the least
significant criteria. The first and most important indicator was
and remains the quality of vulnerability detection.
Appendix A.
Testing Data For Network Scanners In
Various Operating Systems
Due to the large size of this
Appendix (nearly 300Kb), it is available in
a separate file.
|
