Security Testing of Web Browsers

The paper gives an overview of white-box and black-box approaches to security testing.
The paper also describes a black-box fuzzer developed by the authors and discusses the
algorithms used in the tool in a general level. The results of an effort to test web
browsers using this fuzzer are given.

In general the paper is well structured and written in a clear and understandable way.
However, the paper should be proof-read more carefully to turn it into a more polished
state. Now there are multiple confusing sentences, missing words and such.

The topic of the paper is interesting but there seems to be little new in the paper or
anything surprising in the results. Maybe there could be some more insights to the
testing of web browsers. How useful the new generation based fuzzer modules were compared
to the more simple techniques? How many mutations were made to find the 60 bugs? How much
time it takes to generate the mutations, run the test and filter out duplicate defects?
How many previously found/reported bugs did you find? Currently the results section does
not allow the reader to make concusions about the effectiveness of Radamsa, although the
finding 60 bugs does sound impressive to me.

In section 2 you say that the problem of static analysis tools is that they find thousands
of defects. This is quite confusing. Do you mean that the tools report a lot of false
alarms which becomes a problem? (However, you say that a large number of the issues are
real which would indicate that the ratio between true and false positives cannot be that
bad.) Or do you mean that most of the defects do not have security implications or do you
mean something else?

All in all, the paper gives a nice look into security testing and the application of
black-box fuzzers in particular.

The paper "Security Testing of Web Browsers" presents Radamsa, a black-box fuzzer aimed at discovering security vulnerabilities in software. The paper focuses on the application of the tool to web browsers, and discusses the issues related to their security testing. The authors found approximately 60 bugs in commonly used browsers, half of which had some security implications.

In my opinion, the paper is not yet ready to be published. The description of the fuzzer is only qualitative, and offers little detail. The fuzzing methods are listed and described shortly, but even the ones that are presented as novel are described with only a few lines. Moreover, the paper presents no benchmarks or direct comparison against other similar tools, even though it mentions that some are available. It would be nice to have some data on how it compares against these tools with respect to code coverage or the number of security vulnerabilities found.

The paper as a whole also seems a bit unscientific. There are some claims that are not justified or cited, e.g. in section 2 about the requirement to solve unsolved problems in artificial intelligence. It also a bit questionable if the description of bug report handling in the browser industry (section 4) is necessary. These are only examples, I think that the paper needs a fair amount of rewriting so that facts are properly cited and not mixed with the opinions of the authors.

The tool presented in the paper seems interesting, and with the proper amount of work on the above points, there is clear potential for a good publication.

Associate Editor Recommendation: Resubmit after corrections

The paper discusses the security testing of Web Browsers. The anonymous reviewers have discussed the major shortcoming of the paper, which is the lack of detail on the description of the employed methods and their efficiency in fuzz testing. Most of this material is discussed in Sections 3-4. Given these shortcomings I can not directly recommend acceptance but need to request a resubmission.

I also noticed some small fixes the reviewers do not discuss but that should also be fixed:
- The abstract is too short
- Table 2 overflows to the margin

Editor Decision

Your manuscript has been reviewed and reviewers have suggested revising it prior to publication.

There is still a possibility to revise this in due time to get it accepted for publication in the first peer-reviewed issue of the Communications of Cloud Software journal. To achieve this you need to read carefully the comments of the reviewers and update your manuscript accordingly in two weeks (by December 13th). Please, check also the information for authors section providing useful guidelines for revising the paper.

In case you are not able to revise the manuscript by that date, you a later revision will be reviewed for the second issue.

Looking forward for the updated version by 13.12.

Associate Editor Recommendation: Accept the Revised Submission

I have read through the revised version and while not all of the anonymous reviewers comments have been fully addressed, I still find the paper interesting and timely. I do encourage the authors to do a more detailed analysis of the effectiveness of the various techniques employed in the Radamsa tool in some upcoming publication.

Thus I suggest acceptance of the improved revised version.

Editor Decision

Congratulations! Your manuscript has now been accepted for publication in the first peer-reviewed issue of the Communications of Cloud Software journal.