[Buildbot-commits] buildbot ChangeLog,1.432,1.433

[Brian Warner]

Update of /cvsroot/buildbot/buildbot
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv688

Modified Files:
	ChangeLog 
Log Message:
Revision: arch at buildbot.sf.net--2004/buildbot--dev--0--patch-171
Creator:  Brian Warner <warner at monolith.lothar.com>

handle large logfiles without consuming lots of memory

Merged from warner at monolith.lothar.com--2005 (patch 19-25)

Patches applied:

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-19
   Merged from arch at buildbot.sf.net--2004 (patch 159-160)

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-20
   Merged from arch at buildbot.sf.net--2004 (patch 161-166)

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-21
   Merged from arch at buildbot.sf.net--2004 (patch 167)

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-22
   Merged from arch at buildbot.sf.net--2004 (patch 168)

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-23
   Merged from arch at buildbot.sf.net--2004 (patch 169)

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-24
   Merged from arch at buildbot.sf.net--2004 (patch 170)

 * warner at monolith.lothar.com--2005/buildbot--dev--0--patch-25
   handle large log files without using lots of memory


Index: ChangeLog
===================================================================
RCS file: /cvsroot/buildbot/buildbot/ChangeLog,v
retrieving revision 1.432
retrieving revision 1.433
diff -u -d -r1.432 -r1.433
--- ChangeLog	14 May 2005 05:31:30 -0000	1.432
+++ ChangeLog	15 May 2005 23:43:57 -0000	1.433
@@ -1,3 +1,58 @@
+2005-05-15  Brian Warner  <warner at lothar.com>
+
+	* buildbot/status/builder.py (LogFile): revamp LogFiles. Got rid
+	of the old non-offline LogFile, added code to upgrade these to
+	new-style contents-live-on-disk instances at load time (in a way
+	that doesn't invalidate the old Build pickles, so upgrading to
+	0.6.5 is not a one-way operation). Got rid of everything related
+	to 'stub' builds.
+	(LogFile.__init__): create LogFiles with the parent step status,
+	the log's name, and a builder-relative filename where it can keep
+	the contents on disk.
+	(LogFile.hasContents): new method, clients are advised to call it
+	before getText or getChunks and friends. If it returns False, the
+	log's contents have been deleted and getText() will raise an
+	error.
+	(LogFile.getChunks): made it a generator
+	(LogFile.subscribeConsumer): new method, takes a Twisted-style
+	Consumer (except one that takes chunks instead of strings). This
+	enables streaming of very large logfiles without storing the whole
+	thing in memory.
+	(BuildStatus.generateLogfileName): create names like
+	12-log-compile-output, with a _0 suffix if required to be unique
+	(BuildStatus.upgradeLogfiles): transform any old-style (from 0.6.4
+	or earlier) logfiles into new-style ones
+	(BuilderStatus): remove everything related to 'stub' builds. There
+	is now only one build cache, and we don't strip logs from old
+	builds anymore.
+	(BuilderStatus.getBuildByNumber): check self.currentBuild too,
+	since we no longer fight to keep it in the cache
+
+	* buildbot/status/html.py (TextLog.render_GET): use a
+	ChunkConsumer to stream the log entries efficiently.
+	(ChunkConsumer): wrapper which consumes chunks and writes
+	formatted HTML.
+
+	* buildbot/test/test_twisted.py (Parse.testParse): use a
+	LogFile-like object instead of a real one
+
+	* buildbot/test/test_status.py (MyLog): handle new LogFile code
+	(Log.testMerge3): validate more merge behavior
+	(Log.testChunks): validate LogFile.getChunks
+	(Log.testUpgrade): validate old-style LogFile upgrading
+	(Log.testSubscribe): validate LogFile.subscribe
+	(Log.testConsumer): validate LogFile.subscribeConsumer
+
+	* buildbot/interfaces.py (IStatusLogStub): remove
+	(IStatusLog.subscribeConsumer): new method
+	(IStatusLog.hasContents): new method
+	(IStatusLogConsumer): describes things passed to subscribeConsumer
+
+	* buildbot/status/html.py (StepBox.getBox): Don't offer an href to
+	the log contents if it does not have any contents.
+	(StatusResourceBuildStep.body): same
+	(StatusResourceBuildStep.getChild): give a 404 for empty logs
+
 2005-05-14  Brian Warner  <warner at lothar.com>
 
 	* buildbot/test/test_web.py (WebTest.test_logfile): add 5-second