
I agree that both quick and complete tests are needed.  Quick tests should
be run on a daily basis.  How about running complete tests on a
less-frequent schedule, perhaps weekly or monthly, and of course, at least
once before a release.

-G


On 6/11/08 3:10PM , "Kasper Daniel Hansen" <khansen@stat.berkeley.edu>
wrote:

> This is just a followup to what Vince was asking "what would we want
> with unlimited resources.  I agree with others that there are often
> two tiers of tests I think a package could have. Small quick tests are
> (perhaps) sensible to run daily, but longer tests would only have to
> be run once in a while. I however strongly feel that the opportunity
> to get these tests run on a variety of platforms can prove
> indispensable for catching bugs you only see when moving across OS's
> and compilers. Perhaps I have been unlucky but the majority of the
> nasty bugs I have been involved with are of this type.
> 
> So to wrap it up: I am fine with only allowing 5 minutes per run, but
> in my dream world it would be possible for developers to request to
> get the results of a longer test. That will be very nice when (1) you
> are working hard on changing core things in the package and (2)
> nearing a new release when you are wondering about the accumulated
> changes in all packages. By having it being request only, we will
> spend relatively few cpu cycles on it compared to a daily check.
> 
> Of course, I have no great idea as to how such a system could be
> implemented. I assume granting ssh access is out of the question.
> 
> Kasper
> 
> On Jun 10, 2008, at 12:03 PM, Vincent Carey 525-2265 wrote:
> 
>> > This is an important topic.  I believe that the building/checking
>> > process is a key source of value added by the Bioc project for
>> > developers.
>> >
>> > A worthwhile thought experiment: What would be done if we had
>> > unlimited
>> > resources?  It seems to me that a primary resource that a developer
>> > lacks access to is the range of platforms -- hardware, OS, compilers
>> > --
>> > that we want users to be able to work with reliably.  Thus a very
>> > high priority for the build/check system is coverage of the main
>> > varieties of "system" that users are likely to use Bioconductor on.
>> >
>> > The devel branch is very important because it indicates how ongoing
>> > changes to R affect performance/accuracy/interactions of package
>> > code.  Most
>> > developers aren't going to be updating R and all other packages
>> > approximately
>> > daily.  Thus, without the devel build system, many developers would
>> > find themselves working hard when a new R release became imminent, to
>> > port abruptly to the new release.  The devel build system allows this
>> > to occur somewhat more smoothly, and affords the possibility of
>> > feedback
>> > to R core when tentative changes to R are problematic.
>> >
>> > I recognize that the points just made are well-known, and these
>> > remarks
>> > are not made defensively.  Instead I am trying to come up with some
>> > a priori limits to testing requirements falling to bioconductor, as
>> > opposed to requirements that lie uniquely with the developer.  So the
>> > first two priorities are, briefly:
>> >
>> > 1) Cover the platforms
>> > 2) Track performance relative to evolving R
>> >
>> > Now we know full well that our resources require that package testing
>> > be limited to around five minutes.  Is this in itself a value or an
>> > obstacle to ensuring project/package reliability?
>> >
>> > My sense is that this constraint might be a value -- I understand that
>> > a very multifaceted package may have many essential tests that can
>> > finish
>> > rapidly but the sum of testing times exceeds our limits.  Perhaps that
>> > package should be broken up... Perhaps it should get an exception...
>> > I don't know.  Other things being equal, I think it is good for the
>> > software
>> > that there be cases that are legitimate tests that run quickly.  It
>> > means
>> > users can demonstrate the functionality in short order, that the tests
>> > can be varied and examined without long delay, and that, probably, we
>> > have capacity to do more tests of different facets of the package.
>> >
>>> >> From the project perspective, I feel that the tests we need to be
>> > concerned about most involve tests that would indicate problems with
>> > respect to priorities 1) and 2).  Portability problems, or
>> > dependencies
>> > on ephemeral features of R, might only crop up with certain very long
>> > tests ... but I think that would be exceptional.  Thus the deep tests
>> > should be done "at home" and the light ones left for the project
>> > system.
>> >
>> > Finally, the complexity of the project test/build system has to be
>> > kept very manageable -- we have {release, devel} X platform X
>> > {software,
>> > experiment, annotation} and introducing a short/long testing stream
>> > may be feasible but may not pay off.  The point is that even if we did
>> > have a lot more resources I am not sure it would be sound to allow
>> > indefinite test return times.
>> >
>> > I am open to correction or criticism on any point made above.  I am
>> > trying to articulate some points about testing in the project that
>> > are probably quite superficial from the perspective of serious
>> > software
>> > development and engineering -- yet the breadth of testing
>> > accomplished and the necessity of release/devel branches is not very
>> > widely appreciated among some of my contacts in other domains ...
>> > so I have taken this opportunity to articulate these views to the
>> > devel
>> > group.
>> >
>> > The information transmitted in this electronic communica...{{dropped:
>> > 10}}
>> >
>> > _______________________________________________
>> > Bioc-devel@stat.math.ethz.ch mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> 
> _______________________________________________
> Bioc-devel@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> 

-- 
Program Director
Center for Computational Arts, Sciences, and Engineering
Univerisity of Rochester

Tel: 585-273-2794
Fax: 585-276-2097
Email: gregory.warnes@rochester.edu



	[[alternative HTML version deleted]]

