In preparation for detecting flaky tests with BuildPulse, this commit
sets up the rspec_junit_formatter gem to output JUnit XML reports of the
test suite, which is the format used by BuildPulse and various other
tooling that interprets test results.
Because the test suite uses the parallel_tests gem, this commit
incorporates some related changes to make all the parallel_tests gem and
the rspec_junit_formatter gem to cooperate with each other.
rspec_junit_formatter writes everything to a single XML file. That works
fine when there's only one process writing to the file. By default,
whatever process finishes last will write to the file and clobber the
output of all the other processes that wrote to the file. 🙈
To prevent this issue, the parallel_tests wiki recommends adding a
`.rspec_parallel` file to specify its RSpec options
(https://github.com/grosser/parallel_tests/wiki#with-rspec_junit_formatter----by-jgarber),
then the project can specify different files for each process to write
to like so:
--format RspecJunitFormatter
--out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
However, prior to this commit, the Homebrew/brew test suite specified
its RSpec options via the command line. Unfortunately though, there's no
way (AFAICT) to set the equivalent of these options via the command
line:
--format RspecJunitFormatter
--out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
So, we need to use a `.rspec_parallel` file to specify these options ☝️.
However, it appears that RSpec allows you to specify formatters _either_
in an options file (like `.rspec_parallel`) _or_ via command-line args.
But if you specify any formatters via command-line args, then all
formatters in the options file are ignored. (I suspect that's somehow
related to this bit of code in rspec-core:
https://github.com/rspec/rspec-core/blob/v3.10.0/lib/rspec/core/configuration_options.rb#L64.)
With that in mind, in order to have the RspecJunitFormatter configured
in `.rspec_parallel`, we need to move the other formatters into
`.rpsec_parallel` as well, instead of passing them as command-line
args. Therefore, this commit moves all the formatters into a
`.rspec_parallel` file.
This was failing for me locally so I've made some fixes:
- remove the `markdown` flag (as it's the only path now)
- refactor the generation to not use intermediate variables
- discard more weird cases rather than erroring
- exclude @Homebrew changes (e.g. from bots) because we don't care about
these in the release notes
The `--no-publish` flag isn't used anywhere since we removed all the
Bintray code. Also, the `--no-upload` flag used in `pr-pull` doesn't
apply the bottle commit, so its description has been modified.
When running brew commands and interpreting the output, e.g. running
`brew livecheck --json`, it's necessary to stop other programs Homebrew
happens to execute from writing logging output to stdout. Most programs
don't do this, but `bundle install` does seem to.
To reproduce the issue you can run:
```shell
git -C "$(brew --prefix)" clean -ffdx Library/Homebrew/vendor
stdout=$(HOMEBREW_FORCE_VENDOR_RUBY=1 brew livecheck --newer-only --json --cask $(brew --repo homebrew/cask)/Casks/grid.rb)
echo "^^^ was stderr, >>> is stdout: $stdout"
```
If you run it without this change it will print a bunch of output like
this to the stdout before printing out the livecheck JSON output:
```text
Using bundler 1.17.3
Fetching byebug 11.1.3
Fetching coderay 1.1.3
Installing byebug 11.1.3 with native extensions
Installing coderay 1.1.3
Fetching colorize 0.8.1
Installing colorize 0.8.1
[
# Contents of the JSON block.
]
```
With this change the stdout from `bundle install` will be redirected to
brew's stderr, meaning only the JSON goes to stdout, and the rest goes
to stderr.
Formula filenames are required to be lowercase, so the
extract command should downcase version before
using it to construct the filename. I noticed this while
extracting tbb@2020, which has version 2020_U3
and resulted in tbb@2020_U3.rb
Remove all code related to Bintray. It no longer works so there's no
point keeping it around.
Some of this could arguably be deprecated/disabled first/instead but:
I'm not sure I see the sense in keeping stuff around that's known to be
broken.
Run `Homebrew.install_bundler_gems!` a bit earlier for some developer
commands to avoid printing gem installation output in the middle of
normal developer command output.
Alexander Bayandin