Bodhi 101: An Update’s Life Cycle

 

 

This post is part of a series intended to provide those new to Bodhi with an introduction to some Bodhi basics. This post hopes to both introduce the life cycle of an update in Bodhi, as well as to outline the proposed addition of a ‘batched’ request intended to reduce update churn. If you’re a seasoned Bodhi user, you can jump to the Proposed revisions to an update’s life cycle section.

 

An update’s life cycle

First, a user creates a new update or edits an existing update. Updates have plenty of different attributes, but for the purpose of this brief introduction, I will only mention that the updates have a type, severity, status, and request.

The update’s type (i.e., bugfix, security, newpackage, enhancement) classifies the update in relative terms of how it will help the end-user. The update’s severity (i.e., unspecified, urgent, high, medium, low) ranks the importance of getting the update to the end-user. Throughout their life cycle, updates flow through a series of requests and statuses. An update’s status is the current state of the update (i.e., pending, testing, stable, unpushed, obsolete, or processing). An update can request to enter a status (i.e., testing, obsolete, unpush, revoke, stable). If the update meets the requirements for the requested status, it enters that status upon the next mash run.

 

The update begins in the pending state and remains in pending until it requests to be pushed to the testing or stable repository. Once the update is testing, the update will be tested by other users and receive feedback in the form of comments and karma (i.e., -1, 0, +1). The update is eligible to be pushed to the stable repository when it meets the testing requirements (i.e., the update either meets the mandatory number of days in the testing state or reaches the minimum number of positive karma). Note: Additionally, the update can request to be “unpushed”, “revoked”, or “obsolete” from any state, but we will not cover these states here. 

Currently, after the update meets the testing requirements, the next logical step is requesting to enter the stable repository. With autokarma enabled, the update automatically reaches the stable repository upon meeting the testing requirements. With autokarma disabled, the user will be presented with a UI option to push the update to stable upon meeting the testing requirements.

 

The current update life cycle

 

Proposed revisions to an update’s life cycle

In response to requests to reduce update churn, we will add an additional path to the update’s life cycle. This path will essentially gather a batch of updates (namely, all of the updates that have been approved to be pushed to stable) and put this batch in a queue. This batch will request to be pushed to stable once a week. This should result in a smoother experience for end-users, as they will only receive a new batch of updates once a week, as opposed to once a day. This should slightly reduce the mash time six days of the week.

 

The new update life cycle

Instead of immediately requesting to be pushed to stable after meeting the update requirements, the update will instead request to be batched. If autokarma is enabled, the update will immediately be moved to a batched request upon meeting the update requirements. If autokarma is disabled, the user will be presented with a UI option to move the update to a batched request upon meeting the update requirements.

Once the update is sitting as a batched request, it will proceed as before (with the exception of waiting in a queue for up to seven days). If the packager should believe the update would benefit from an immediate request to stable, (s)he will now have the option to immediately request stable (and thus bypassing the batched request) via a UI option. Otherwise, once a week, a cron job will execute a script setting the request of all updates currently requesting batched to request stable.

If the update’s severity is urgent or if the update’s type is newpackage, the update will bypass the batched request and immediately request to be pushed to stable. Urgent updates need to reach end users as soon as possible, so holding them in a queue wouldn’t be beneficial. Additionally, newpackage updates already tend to have a long process time, and we do not want to further slow their process time.

 

The current PR can be found here. Feel free to leave any feedback on the PR or as a response to the email with the subject “Two more concrete ideas for what a once-yearly+update schedule would look like” on the Fedora dev mailing list.

 

 

Advertisement

Outreachy: Bash Completion

Prior to applying for Outreachy, one of the points that drew me to Fedora’s Bodhi project was the proposed idea of bash completion for the Bodhi command line interface (CLI). For those unfamiliar with bash completion, it is the phenomenon that occurs when you begin to type a command into your Unix shell (in this case: bash), press tab, and bash either (1) autocompletes the command for you or (2) provides a list of all of the possible ways to complete that command. During my undergrad CS studies, my favorite projects entailed creating Unix shells, command line chat servers, and text-based command line games. All of these projects have a clear common denominator: the command line. I found that I would spend hours upon hours blissfully trying to perfect my CLI’s user interface. Thus, this seemed to be the Outreachy project of my dreams!

To implement Bodhi’s CLI, Bodhi uses Click, which is a cool Python package used to create CLIs. If you aren’t very familiar with Bodhi, the majority of the code is written in Python. Prior to my internship, I was not well-versed in Python let alone Python libraries or Python packages (I had only ever used it to write small scripts, test algorithms, and create servers).

Because of the nifty framework Click, the bash completion for Bodhi proved simpler (far simpler) than I anticipated. Click actually already supports a certain level of bash completion for its CLIs. All that was required was to generate the activation script and to get around the Python errors. Bodhi now ships with the bash-complete activation script, so all you need to do is press tab! The PR enabling bash completion can be found here.

 

What does Bash completion look like within the Bodhi CLI?

Bash completion supports the following commands:

• bodhi
  • updates
    • subcommands: comment, download, edit, new, query, request
      • each subcommand’s respective parameters (too many to list)
  • overrides
    • subcommands: edit, query, save
      • each subcommand’s respective parameters (too many to list)

 

Enabling Bash Completion

It is enabled by default. Just type the following* into your Vagrant instance’s shell:

bodhi <TAB><TAB>

*Replace “<TAB><TAB>” with two quick taps of the tab key ;). Oh, and yes — there’s a much-needed space in between bodhi and <TAB>!

 

Uh oh! Why am I seeing loads of warnings?

For bash completion to work properly (i.e., without various Python errors printing to your screen each time you press the tab key), you will need to ignore Python warnings. Otherwise, your output will look like this:

 

Ignoring Python warnings

Python warnings tend to be helpful in a dev environment, so proceed at your own risk of future frustration. You can either ignore all Python warnings, or you can temporarily ignore Python warnings while using bash completion.

To ignore all Python warnings, the user could change the following line in their .bashrc:

export PYTHONWARNINGS="once"

to

export PYTHONWARNINGS="ignore"

Again, as a dev, you probably don’t want to ignore Python warnings everywhere. It may be a good idea to only ignore the warnings within the Bodhi CLI. If you’d still like to see the Python warnings in outside of the Bodhi CLI, you can use the following (somewhat hacky) solution by editing the bodhi-complete.sh file:

 

Scalability and future improvements

Earlier I mentioned that Click only supports a certain level of bash completion. Click has yet to implement the ability of dynamic completion. Thus, because Bodhi relies on Click for its CLI and thus bash completion, Bodhi also has yet to implement the ability of dynamic completion. This feature would look like this:

bodhi updates new bodhi-2.7.0-<TAB>

where the auto complete options would result in the following

bodhi updates new bodhi-2.7.0-
fc25 fc26

Though, for Bodhi to use dynamic completion, it would require changes to upstream Click. Unfortunately, I cannot find evidence that Click has plans or desire to add this feature anytime soon.

 

Reflection and lessons learned

After learning how simple it is to create CLIs with Click, I am inspired to port some of my favorite projects (originally written in C) to Python. Although I am still discovering many of its nuances, Python is quickly becoming my favorite programming language. I still have some leftover habits of writing Python as though it’s C and thus may occasionally find myself not realizing a simpler way of approaching a certain line of code, but that’s part of what makes learning Python great: consistently finding yourself shocked at its simplicity. And lucky for me — I have a very helpful mentor!

Keep in mind:

1. There may be a simpler way of adding a new feature than you initially imagined, so before coding, be sure to search for external packages, tools, and libraries — and most importantly: check with your mentor!
2. An Outreachy project may not be exactly what you anticipate — and that’s okay! — so it may be a good idea to vest interest in more than one project idea.
3. You don’t have to be extremely well-versed in an open source project’s main language (though it will certainly save you quite a bit of time)
4. If you want a feature in open source project A which depends on open source project B, it may require contributing to open source project B, which could cause hiccups.

 

Many thanks to my very helpful, patient, and encouraging mentor, bowlofeggs!

Outreachy Round 14

I am both excited and humbled to announce that I was accepted into Outreachy Round 14! I will be working with my mentor Randy Barlow (bowlofeggs) to improve Bodhi: the web-system that publishes updates for Fedora. We will hash out the plans in greater detail next week, so I’ll post an update after that meeting. As of now, the plan is to make improvements to the Bodhi CLI — more specifically, command completion.

If you’re thinking, “Okay, wait — what’s Outreachy?” Let me give you a brief summary: Outreachy is a wonderful internship opportunity for underrepresented groups in FOSS. Its primary goal is to encourage underrepresented groups to become FOSS contributors. An Outreachy participant is paired with a kmentor (supplied by the FOSS organization) to work on a project. The projects range from Linux Kernel development to designing a coloring book. There’s truly a project for everyone. Outreachy is hosted by the Software Freedom Conservancy, with help from RedHat and the GNOME Foundation. If you’re interested in learning more–especially if you’re a member of an underrepresented group in FOSS–you can find more information here.

How I came to apply to Outreachy
I’ve been mustering up the courage to apply for Outreachy for over a year. During rounds 11-13 I ended up with intense feelings imposter syndrome a day or two before the deadline, and thus never submitted an application before Round 14. This round, I started and powered through the application tasks only a few days before the deadline (no time for imposter syndrome to set in). I reminded myself that It’s only impossible to get in if you don’t apply and that internships are meant to be learning opportunities. I held my breath through the application process and got in. (I have to keep reminding myself because I’m still in shock, :D.) I will outline my initial contributions (i.e., application tasks) below.

Initial contributions
My first contribution to Fedora was also my first FOSS code contribution. This contribution consisted of fixing a bug in Bodhi. When an update reaches the required time in testing, Bodhi comments to alert the user that it can be pushed. The user can edit this update anytime, including after Bodhi comments that the update can be pushed. After a build is updated, the karma counts and the number of days in testing are reset, and thus the ability for the update to be pushed is also reset. If the karma or days in testing threshold is reached, the user should yet again be able to push the update. Bodhi should comment to notify the user of the recent ability to push. The error occurred when Bodhi did not post a comment notifying the user that the update could be pushed if Bodhi already commented that the update could be pushed prior to the karma (and thus, days in testing) reset. This error occurred because the method met_testing_requirements() determined whether the testing requirements were met based on Bodhi’s push approval message prior to the recent edit. I fixed this error by creating a new method to only retrieve the comments since the most recent update. The methods that determine the amount of karma and whether the testing requirements were met used the new method I created instead of simply retrieving all comments. I also wrote a test to ensure that this type of error won’t occur in the future. My initial contribution can be found here: https://github.com/fedora-infra/bodhi/pull/1396. The commit can be found here: https://github.com/fedora-infra/bodhi/commit/ceaf533d9a2dd0dd3496a2c12030689956f453a7.

My second code contribution was also to Fedora’s Bodhi. This contribution entailed sharing the –notes-file parameter between the new and edit CLI commands (i.e., a new update and an edited update). This allows the user to specify a file in which the update’s notes are contained. First, this required adding the option of using the –notes-file parameter in the CLI. Now that the user can use –notes-file as a parameter, the –notes-file parameter must actually retrieve these notes from the notes file. In order to achieve this, the file would need to be opened, read, and placed into the notes variable. This required making a function to get and/or process the notes, which was aptly named “_get_notes”. Such a change also required substantive testing. I created a test to ensure that the –notes-file parameter would be properly handled when creating an edit request. I created another test to assert that providing both the –notes-file and the –notes parameters to an otherwise successful updates edit request would result in an error. Finally, I placed a note in the man pages concerning this change. You can find this contribution here: https://github.com/fedora-infra/bodhi/commit/305f9aa8663ede691ccdd10cef005124276cbd21.

My most recent contribution was my third contribution (also to Fedora’s Bodhi). I have ensured that when an exit code of a process is zero, the standard error (if any) will be logged at the info level on the bodhi logger. If the exit code of a process is non-zero, the standard error will be logged at the error level on the bodhi logger. This was in response to a bug that resulted when emails were sent containing standard error output of processes with a zero exit code. Note that standard error does not always indicate an error has occurred — only when there is a non-zero exit code can we be sure that an error has actually occurred. I also wrote tests to ensure that this bug will never reoccur. This contribution can be found here: https://github.com/fedora-infra/bodhi/pull/1417.

General advice to prospective interns as an incoming intern:
If you’re a prospective Outreachy applicant, and if you find the application process a bit daunting, you’re not alone. But hey, you’ve got this! Take a few deep breaths and remember: the only way you absolutely cannot get in is if you don’t apply! Your mentors and the community are there to help you, so don’t be afraid to ask any questions. The program is aimed at newcomers who want to make their first contributions, so please remember that if you begin to feel inadequate in any way. I know that it can feel impossible if you start comparing yourself to the people who have committed over a million lines of code to a project. Always remember that they had to start somewhere!

If you have any questions about my experience with the application process, want some pointers or just a confidence boost, PLEASE don’t hesitate to contact me. If I can get accepted, you can too!