Release Insights
new
Release small, and frequent.
Now with Athenian you can't only measure your release cycle time but also understand its frequency. In particular if you have a service based architecture that depends on multiple releases, understanding your release frequency per repository will be useful to help you understand the true velocity at which you're moving and where potential bottlenecks are.
Screen Shot 2020-09-03 at 2
Understanding what is inside a release, how large they are (in terms of # of PRs or LoC impact) and how many contributors were involved is now possible from the new release table.
Screen Shot 2020-09-03 at 2
Code Reviews Per Repository
new
Code Reviews are invitations to share knowledge amongst the team and give feedback. This latest chart allows you to understand where code is being reviewed and where it's not yet.
We do this by breaking down repositories in reviewed and not reviewed Pull Requests.
We also provide you with a top level metric of "Pull Requests Reviewed (%)" which you can use to set internal goals, such as
"100% of Pull Requests reviewed on our web application, and an average of 80% of PRs reviewed across our whole organization".
Screen Shot 2020-08-28 at 9
As always you can jump to the Pull Request table and filter on "Not Reviewed" to see where you're skipping this best practice.
Screen Shot 2020-08-28 at 9
Or choose to filter on your offending repositories to understand why certain pull requests are being merged before they are reviewed.
Distribution Charts & Code Bypassing Pull Requests
new
Hot on the announcement of 4 new charts earlier this week we bring 2 new updates today.
We got valuable feedback from our users that while their Cycle Time across each stage is useful to quickly spot where the bottlenecks in their Software Delivery Pipeline are, it often prompts the question:
How do I know if my Average Cycle Time isn't dominated by outliers?
In the coming months you'll see more features being released that help you as an engineering leader to identify outliers (we released the default option to remove stalled PRs back in June). Which is why we're introducing Distribution Charts.
Distribution Charts
Now your Cycle Time in each stage (WIP, Review, Merge and Release) and your Lead Time, no longer just show their trend over time but also provide a distribution.
Screen Shot 2020-08-07 at 2
On the y-axis you find the # of Pull Requests and on the x-axis you find your PRs bucketed based on the time spent in each stage. This is done on a logarithmic scale so you can quickly spot your outliers.
Let me give you a great example where this is very apparent. The following company uses CI/CD across their repositories and almost always releases in less ±1 minute, but it's Average Cycle Time in their Release stage is 24 hours. With the distribution chart you can now see that while 97% of their PRs release in ±1 minute, a small number of PRs end up being significant outliers:
Screen Shot 2020-08-07 at 2
Screen Shot 2020-08-07 at 2
As an engineering leader, it's the outliers here you want to focus on and understand what happened there, and to either choose to ignore it or to adopt your practices or tooling to avoid these.
Code Bypassing Pull Requests
Athenian's users are diligent about using Pull Requests. However at times there are a hidden issues in our Software Delivery Pipeline that stops engineers from using PRs and instead commit directly. A common example is that locally all the tests passed and the engineer finds himself waiting for a long running CI check when they open a PR, so instead they choose to commit directly.
As engineering leaders we want to discover
"Why are we committing directly without a Pull Request?
, this new table in the Work In Progress section shows you the repositories where this is most common and allows you to inspect the offending commits.
Screen Shot 2020-08-07 at 7
More Pull Request Insights
new
You spoke, we listened!
We're releasing 5 new charts related PR size and activity.
Pull Request Size Insights
Small PRs, released often are the corner stone of high performance engineering teams. Till date we gave you insight into your avg. PR size and the ability to order your Pull Requests on size in the table. From our conversations with our users we learned that you'd like to be able to dig in even further and understand how you're doing in terms of PR size.
We've therefore introduced 3 new charts into the Work In Progress section.
  1. Distribution of # PRs based on lines of code, a left skewing distribution and few PRs in your long-tail is what you should be aiming for.
  2. Breaking down your PRs in 5 size buckets, it's the 500+ lines one you should aim to avoid whenever possible.
  3. The ability to dig into the largest PRs during the date range you've selected and their average lead team.
pr-size-july-2020
Pull Request Activity
You're now able to look at the quantity of Pull Requests broken down by date, repository and author. Quantity metrics are useful to help you understand where your biggest areas for improvements are. Be careful though to never use quantity metrics as a way to rank individuals, they do not work for that.
pr-activity-july-2020
Excluding stalled pull requests, 5x performance improvement, updated stage badges and a new volume chart
new
improved
Excluding stalled pull requests
Stalled pull requests are PRs that have had no activity for a long period of time.
For many organizations using Athenian their product experience was biased by seeing their backlog of stalled pull requests included in the charts, pull requests section and stage metrics.
These PRs take up your attention when most of the time, they don’t really influence the current work being done.
Being able to review stalled PRs is important because they usually do require an action, either to be closed, rebased, or picked up to work on again.
To solve this issue, and offer a lightweight user experience, we’ve by default excluded stalled pull requests and decided to include a checkbox in the calendar that allows you to include them.
449D0439-70C6-4603-87AD-F6A22483A3F8
How does it work?
By default, any pull requests that had any activity (created, reviewed, merged, released etc.) in the date range you selected will be included. If you choose to select “Include stalled pull requests” it will also include all pull requests that are open but didn’t have any activity during this period.
Performance improved by 5x in the last 2 weeks
During the last two weeks our team has worked hard to improve the loading time of our product. During this period we optimized indexes on the database, added new layers of caching, and continued the work on our transition from live computation to precomputed data. Important to note here is that even with caching and precomputed results your data is never older then 5 minutes.
Updated stage badges
Before this release the stage badges would should the number of pull requests that completed the stage. After observing how people are using Athenian we’ve decided to change them to be the number of pull of requests currently in that stage.
When you now see the number 4 here in this review stage, it means there are currently 4 pull requests that have started the review process but haven’t yet completed it.
2F9C99A0-C6B1-47DA-B3E1-6B5D96BA0A46
When should you use this?
If you want to have a quick glance on how much work is pending in each stage.
New Volume Chart
We’re starting to add some ‘simple’ volume charts that will help you put in context other metrics (see Speed, Volume, Quality, and Impact). The first one we added is the # of pull requests created.
6761243B-32F2-4FA2-BEB5-18C65D76BEFC
Contributor filter is now more useful
improved
Today we released an important update that changes the behavior of the contributor filter. Before, if you had a contributor selected, the metrics would be calculated over all of the pull requests they were involved in. No matter if they were a reviewer, merger or author. Now when you filter on contributors, it will only include pull requests (and calculate their metrics) from which they are an author.
The motivation behind this change is that we wanted to allow teams to be fully in control over improving their metrics, now by only including pull requests which the team authors, this makes that possible.
Pro-tip:
this change is particularly useful when you want to exclude pull requests authored by bots.
Screen Shot 2020-06-02 at 5
new
It's finally here! You're now able to define your teams in
Settings
and filter on them using the contributor filter.
Important to know:
  • Any user is allowed to create, edit and delete teams
  • One contributor can be part of multiple teams
  • Anyone who is not a member of a team will be shown as "Other" in the contributor filter
Pro-tips:
  • Create a team for your bots, allowing you to easily exclude them
  • If you have an open-core repository, add all your company's team members to teams and use 'Other' to understand your community metrics
Contributor filter with teams
Screen Shot 2020-05-26 at 8
Frontend team selected
Screen Shot 2020-05-26 at 8
Settings
Screen Shot 2020-05-26 at 8
Release Settings
new
We have released ‘release settings’ allowing you to customize what “released” means for each repository. There are 3 modes you can pick from:
  1. Automatic
  2. Branch
  3. Tag
Automatic
Athenian will detect if any tags were present during the time period you selected and if they are, will set the repository release strategy to “when a new tag is published”. If they are not, it will use the repositories default branch.
Branch
You can select a specific branch such as
master
or
production
or you can use a regular expression pattern to match multiple branch names.
Tag
You can set it to consider
any
tag a release or use a regex pattern to for instance match on tags starting with
v
for version.
F6FEF697-0FEF-45D9-BE45-868F759C6619
Once you’ve updated your release settings, all metrics based on it, such as Lead Time will update. You’ll also see an update of the “released” label in your Pull Requests table.
This setting can be found by going to your profile picture > Settings > Releases.