New MQ option to Buildbot
Short description: As of version 0.9.0, Buildbot uses a message-queueing structure to handle asynchronous notifications in a distributed fashion. The current Message Queue works but have some problems, from costing to function, from availability to race condition. According to community’s discussion, I’ll implement a new MQ plugin and try to solve the data-ordering problem.
- Name: Wei Wang
- Email: email@example.com
- IRC: MatheMatrix
I propose to implement a new Message Queue plugin to solve these current problems:
- Now Buildbot use the mq which is implemented by dustin, this costs a lot and will draw away our attention, we need to focus on Buildbot, not message queue.
- The current message queue uses only one exchange, so all message will send to a single exchange.
- The current message queue is not very reliable, it doesn’t have a retrying mechanism for status updates (see ticket #1733).
- The MQ layer doesn’t support other Message Queue like ZeroMQ or others.
- The callback may be invoked a second time before the Deferred from the first invocation fires.
- Message ordering can’t be guarantee.
- Many components are missing.
I’ll try to develop a new plugin to solve these problems,mainly based on Kombu. Besides, I’ll do some research to solve those complex problems which need some CS theory.
The current message queue uses a particularly simple architecture: in AMQP terms, all messages are sent to a single topic exchange, and consumers define anonymous queues bound to that exchange. It implemented in the mq dir, including base.py, simple.py and connector.py. Now we use MQConnector.produce and MQConnector.startConsuming as API and used frequently. And the MQConnector’s UML class diagram just like this:
Once Buildbot master runs, the BuildMaster.create_child_services will create self.mq using mqconnector.MQConnector and then this mq will always available as self.master.mq.We use mq in buildrequest and many other cases, so I’ll try to make the new plugin compatible to the current situation. Here is all files which use mq explicitly:
Kombu is a messaging library for Python providing an idiomatic high-level interface for the AMQ protocol, and also provide proven and tested solutions to common messaging problems.it is easy to add support for non-AMQP transports and there is already built-in support for Redis, Beanstalk, Amazon SQS, CouchDB, MongoDB, ZeroMQ, ZooKeeper, SoftLayer MQ and Pyro.
Kombu support various exchanges, Consistent exception handling and many other features, I will have future discussion with my mentor to decide what we need, how to package Kombu and details of APIs which we provid.
First of all, I’m planning to devote 2 to 3 hours each day in the GSoC period to this project on average. I’m certainly willing to make more efforts if the project is behind schedule. I’m pretty sure I have no other commitments during this summer.
The final deliverable would consist of:
- A written specification of the MQ layer that is to be followed throughout the development.
- A new plugin that works well.
- Make sure all current code will work well under new MQ layer.
- Test cases.
Staring on April 21(UTC), when Google announces the accepted students, the whole available time of GSoC is about 16 weeks. Accordingly, my planned schedule:
- Week 1 – Week 3: Get to know the community, including blueprints track, code review, code styles and so on. Discuss MQ layer details with my mentor.
- Week 3 – Week 5: Document the MQ plugin, be more familiar with BuildBot and Kombu, write unit tests if time allowed.
- Week 5 – Week 7: Implement the MQ plugin, debug and find issues.
- Week 7 – Week 9: Make sure the new MQ layer can work with current code. Do some tests including performance and availability.
- Week 9 – Week 10: Try to solve data-ordering problem and document.
- Week 10 – Week 13: Implement the solution and do test.
- Week 13 – Week 15: Handling unforeseen problems if necessary, update other part’s code to use some advanced feature implemented in MQ layer.
- Week 15 – Week 16: Tidying up any loose ends and do summary.
Open Source Development Experience
I use Linux and benefits form open source community for years. I love open source and have put some of my work to Github and hope it can be helpful to anybody(an animation example based on pygame which can be used in wxPython, a shell to install OpenStack conveniently ).
From last year, I did contribution to OpenStack community and here is my profile in Launchpad (OpenStack community’s bug track and blueprints track all use it): https://launchpad.net/~damon-devops. I mainly filed and fixed bugs for various projects.
Besides, I’m active at OpenStack’s dev and general mail-list, and posted a thread to discuss this project(MQ option): http://sourceforge.net/p/buildbot/mailman/message/32104525/
I have experience of two internships:
- Earthquake Resistance and Disaster Reduction Institution,Beijing University of Technology (June 2012 – October 2012): Develop a bridge’s detection system, use C# and SQL Server.
- Earthquake Resistance and Disaster Reduction Institution,Beijing University of Technology (June 2013 – October 2013): Develop a health monitor system, mainly use wxPython, matplotlib, Pygame, zeromq, Multi-thread.
- Institue of Computing Technology, Chinese Academy of Science (January 2014 – March 2014): Develop and Deploy OpenStack, mainly focus on Neutron.
I now study at ChangChun University, School of Science, as a junior undergraduate student, and my GPA is 3.3/4.0 top 10% at my Dept.
I have installed Buildbot on my computer and tested the pyflakes, I’ll do more and test other projects these days.
I have some experience about mq and distributed system. I’m building a CI system of lab which I work at, once I know Buildbot, I find it’s very interesting and community is really friendly, besides, I want to learn some knowledge of software engineering according to Buildbot. What’s more, Buildbot servers many projects, this attracting me, too.