The `PUPPETEER_SKIP_DOWNLOAD` environment variable is used to prevent Puppeteer from automatically downloading browser binaries.
It was removed in [56926] due to the migration to Playwright. However, because of the QUnit tests, Puppeteer is actually still a dependency.
Until those tests change, we have to keep this environment variable to prevent unnecessary downloads on CI.
Props SergeyBiryukov.
Fixes#59517.
git-svn-id: https://develop.svn.wordpress.org/trunk@56954 602fd350-edb4-49c9-b593-d223f7449a82
This is a follow-up to [56926], where this logic has been reintroduced after it was originally removed in [56198].
Props desrosj.
See #59517.
git-svn-id: https://develop.svn.wordpress.org/trunk@56942 602fd350-edb4-49c9-b593-d223f7449a82
As per the migration plan shared last year, this migrates all browser-based tests in WordPress core to use Playwright.
This includes end-to-end, performance, and visual regression tests.
Props swissspidy, mamaduka, kevin940726, bartkalisz, desrosj, adamsilverstein.
Fixes#59517.
git-svn-id: https://develop.svn.wordpress.org/trunk@56926 602fd350-edb4-49c9-b593-d223f7449a82
Following [56829], the previous `timeout-minutes` value of `5` is insufficient when approaching 10 retries.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56830 602fd350-edb4-49c9-b593-d223f7449a82
This increases the number of times to retry restarting a workflow from 2 to 10. Retries use exponential backoff to space out retries.
In most cases, only 2 retries was sufficient. However, there are occasionally scenarios where the original workflow is still running and cannot be restarted, resulting in an error. This typically happens during periods of ongoing service degradation, or workflows with a significant number of jobs (GitHub Actions is sometimes very slow to mark these as finished).
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56829 602fd350-edb4-49c9-b593-d223f7449a82
[56780] changed Slack failure notifications to not send a failure notification during the first run of a workflow. Because workflows automatically restart once, a failure during the first run can be ignored.
This adjusts the workflow to also not send a “fixed” notification when the second run of a workflow succeeds after a failure. Because the original failure is no longer reported, these notifications are unnecessary.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56827 602fd350-edb4-49c9-b593-d223f7449a82
This was causing the version of WordPress being tested to be listed as `latest` even when that is not the case.
Props davidbaumwald.
See #58977.
git-svn-id: https://develop.svn.wordpress.org/trunk@56823 602fd350-edb4-49c9-b593-d223f7449a82
Because the contents of binary files are not included in patches by default, occasionally empty versions of these files are committed on accident. This is the most commonly happens within default themes because they usually contain image and font files.
This adds a job to the theme related GitHub Action workflow that checks for the presence of zero-byte files so that these instances are caught more easily.
Props costdev, hellofromTonya, luminuu, onemaggie.
Fixes#59527.
git-svn-id: https://develop.svn.wordpress.org/trunk@56792 602fd350-edb4-49c9-b593-d223f7449a82
After [53947], all workflows will automatically be restarted once in an attempt to rule out reasons for failures, such as timeouts or network hiccups.
Second attempt at [56404], which was previously reverted in [56407].
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56780 602fd350-edb4-49c9-b593-d223f7449a82
This adds Twenty Twenty-Four to the Default Theme test workflow in GitHub Actions.
Follow up to [56764], [56726], [56721], [56716].
See #59447.
git-svn-id: https://develop.svn.wordpress.org/trunk@56770 602fd350-edb4-49c9-b593-d223f7449a82
PHPCS can now be run on the latest PHP version as all known PHP 8.x compatibility issues (in WPCS) have been fixed.
Follow-up to [49162], [56695].
Props jrf, desrosj.
See #59161.
git-svn-id: https://develop.svn.wordpress.org/trunk@56696 602fd350-edb4-49c9-b593-d223f7449a82
This introduces GitHub Action workflows for testing the installation and upgrade process to ensure there are no unexpected issues. The goal of these workflows is to replicate some of the manual testing that happens during the release process. This does not replace that testing which is a fantastic, low barrier way for someone newer to contributing to get involved. Instead, it should serve as an additional sanity check to ensure no combinations are missed.
The installation workflow tests installing the latest or specified version of WordPress on all PHP and MySQL version combinations for both single and multisite.
The upgrade testing will test upgrading from each branch that still receives security updates to the latest or specified version on all PHP and MySQL version combinations for both single and multisite.
Both workflows confirm that no errors are encountered when using WP-CLI to install or upgrade WordPress. No test assertions of any type are currently performed, but adding some will be explored.
Props costdev, azaozz, ironprogrammer, desrosj.
See #58977.
git-svn-id: https://develop.svn.wordpress.org/trunk@56661 602fd350-edb4-49c9-b593-d223f7449a82
This simplifies the logic within the `slack-notifications` and `failed-workflow` steps in GitHub Action workflows to use the `contains()` function and object filtering.
This makes it simpler to perform the needed checks by removing the need to list out every single dependent job defined in `needs`.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56660 602fd350-edb4-49c9-b593-d223f7449a82
This changes the environment variable used in GitHub Action workflows to skip downloading the browser binary that’s a peer dependency when it’s not needed.
In [56647], the version of `puppeteer` peer-dependency was bumped to `>= 20.0.0`. Starting in version `20.0.0`, puppeteer switched to using Chrome for testing instead of Chromium. With this release, the `PUPPETEER_SKIP_CHROMIUM_DOWNLOAD` environment variable was removed in favor of the more generic `PUPPETEER_SKIP_DOWNLOAD`.
All workflows that do not need a browser binary now contain the correct `PUPPETEER_SKIP_CHROMIUM_DOWNLOAD` variable.
Follow up to [56647].
See #58863.
git-svn-id: https://develop.svn.wordpress.org/trunk@56659 602fd350-edb4-49c9-b593-d223f7449a82
This updates all 3rd-party GitHub actions to their latest versions. The following actions were updated:
- `actions/cache`
- `actions/checkout`
- `actions/setup-node`
- `codecov/codecov-action`
- `shivammathur/setup-php`
- `slackapi/slack-github-action`
In the latest version of `actions/checkout` (`4.0.0`), a new input was introduced to control the output of command progress. This change uses this new `show-progress` input to turn off displaying progress by default. Progress will be shown when a workflow is run with debug mode enabled, just in case it contains helpful information.
Props johnbillion, desrosj.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56537 602fd350-edb4-49c9-b593-d223f7449a82
This extends the performance test workflow added in [55459] to also run tests against the target branch (if running on a pull request) or the previous commit (if running on trunk).
Those results are then compared with the ones from the current commit, and the difference is displayed as a GitHub Actions workflow summary for convenience.
Props mukesh27, flixos90, desrosj, joemcgill, swissspidy.
Fixes#58358, #58359.
See #56150.
git-svn-id: https://develop.svn.wordpress.org/trunk@56506 602fd350-edb4-49c9-b593-d223f7449a82
PHP 8.3 is expected to be released at the end of November 2023.
Enabling the tests to run in CI on PHP 8.3 allows WordPress core to start getting ready.
Note: Xdebug-related tests will not be run on PHP 8.3 at this time as the Docker image for PHP 8.3 does not contain Xdebug yet. Once a stable release of Xdebug 3.3.0 is available, it can be added to the Docker image and the test step can then be enabled for PHP 8.3.
Follow-up to [53922], [56492], [56495].
Props jrf, costdev.
See #59231.
git-svn-id: https://develop.svn.wordpress.org/trunk@56498 602fd350-edb4-49c9-b593-d223f7449a82
This ensures the Hosting Test results API key is present for reporting results.
Follow up to [56439] and [56440].
See #30462.
git-svn-id: https://develop.svn.wordpress.org/trunk@56443 602fd350-edb4-49c9-b593-d223f7449a82
This modifies the PHPUnit workflow to add MySQL version 8.0 to the strategy matrix. It also adds an additional strategy matrix to expand testing to include several supported MariaDB versions.
To prevent repeating code, the business logic of the PHPUnit testing has been moved to a new, callable workflow.
Because callable workflows target a specific branch, branches created in the future will continue to benefit from improvements made to the workflow in `trunk` without requiring backports. If a breaking change is required, older branches will need to be updated to target the commit SHA representing the final commit of the previous state.
Props johnbillion, pento, boonebgorges, netweb, nacin, desrosj.
Fixes#30462.
git-svn-id: https://develop.svn.wordpress.org/trunk@56439 602fd350-edb4-49c9-b593-d223f7449a82
This reverts [56404], [56405], [56406].
More testing is needed to ensure multiple messages are not sent unintentionally.
Unprops desrosj.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56407 602fd350-edb4-49c9-b593-d223f7449a82
This adds `startup_failure` to the list of conclusions that the Slack workflow looks for when determining whether to send a failure notice.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56406 602fd350-edb4-49c9-b593-d223f7449a82
[53947] introduced a callable workflow that allowed a workflow run to be retried automatic. By default all workflows are retried once.
Since a run is not considered “failed” until after the automatic retry, the first Slack message is unnecessary and can cause a lot of noise when there are network hiccups.
This alters the logic to skip a failure notice in Slack until the second failure.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56404 602fd350-edb4-49c9-b593-d223f7449a82
Because `slack-notifications` job requires all preceding jobs in each workflow to complete, there’s no need to include the same jobs in the `needs` configuration for the `failed-workflow` one.
See #58867.
git-svn-id: https://develop.svn.wordpress.org/trunk@56402 602fd350-edb4-49c9-b593-d223f7449a82
This sets the following environment variables when running automated performance tests to avoid side effects that can skew performance data when the default development environment variables are used:
* `SAVEQUERIES`: `false`
* `SCRIPT_DEBUG`: `false`
* `WP_DEBUG`: `false`
* `WP_DEVELOPMENT_MODE`: `''`
Props rajinsharwar, desrosj, mukesh27, joemcgill.
Fixes#58825.
git-svn-id: https://develop.svn.wordpress.org/trunk@56313 602fd350-edb4-49c9-b593-d223f7449a82
This restores the GitHub Actions job responsible for automatically retrying a failed workflow once within the E2E testing workflow.
[56198] disabled Slack notifications for this workflow because of the increased number of timeout errors occurring after recent changes until they could be further investigated. Even though the signal-to-noise ration was way too high, there’s still benefit in retrying the workflow once to see if the timeout can be resolved without human intervention. The one retry attempt will not result in any Slack notifications.
Follow up to [56198].
See #58779.
git-svn-id: https://develop.svn.wordpress.org/trunk@56308 602fd350-edb4-49c9-b593-d223f7449a82
Remove Slack notifications of E2E test suite failures while the tests are flakey to improve the signal-to-noise ratio.
Fixes#58779.
git-svn-id: https://develop.svn.wordpress.org/trunk@56198 602fd350-edb4-49c9-b593-d223f7449a82
This also removes the random compat library which is no longer necessary, and adjusts unit tests and CI workflows that no longer need to take PHP 5.6 into account.
Thank you for your service, PHP 5. Onwards!
Props SergeyBiryukov, mukesh27, dingo_d, audrasjb, jrf, costdev, azaozz, JavierCasares, hellofromTonya, samiamnot, spacedmonkey, masteradhoc, knutsp, garyjones, chanthaboune
Fixes#57345
git-svn-id: https://develop.svn.wordpress.org/trunk@56141 602fd350-edb4-49c9-b593-d223f7449a82
This adds a matrix to the e2e test runs on GitHub Actions so the tests run both with and without `SCRIPT_DEBUG` enabled. This allows us to catch scenarios where the minified and non-minified files may differ.
Props Clorith, joemcgill
Fixes#58661
git-svn-id: https://develop.svn.wordpress.org/trunk@56113 602fd350-edb4-49c9-b593-d223f7449a82
Modify the E2E workflow to store the screenshots and HTML files generated during failing tests as an artifact in the workflow.
No artefact is stored for passing test runs.
Props peterwilsoncc, isabel_brison.
Fixes#58596.
git-svn-id: https://develop.svn.wordpress.org/trunk@55991 602fd350-edb4-49c9-b593-d223f7449a82
Update the performance tests to account for the relocation of the theme unit test data repository to the WordPress organisation on GitHub.
Props costdev, kafleg, williampatton, dd32, otto42, poena, jrf, joemcgill, peterwilsoncc, mukesh27.
Fixes#58569.
git-svn-id: https://develop.svn.wordpress.org/trunk@55976 602fd350-edb4-49c9-b593-d223f7449a82
The `permissions` key in a job declares the GitHub permissions that are granted to the token that's used by the job. Restricting the permissions reduces the impact that a vulnerability in the CI system can have.
Props desrosj, johnbillion
See #57865
git-svn-id: https://develop.svn.wordpress.org/trunk@55715 602fd350-edb4-49c9-b593-d223f7449a82
This adds the new performance measurement workflow introduced in [55459] to the one that tests old branches on a regular schedule.
This ensures it continues to work as expected over time, even if the branches with this workflow are not updated for a bit of time.
Props flixos90, joemcgill, desrosj.
See #57687.
git-svn-id: https://develop.svn.wordpress.org/trunk@55507 602fd350-edb4-49c9-b593-d223f7449a82
This updates three GitHub Actions to their latest versions:
- `shivammathur/setup-php`
- `actions/cache`
- `wow-actions/welcome`
The welcome action now uses the `GITHUB_TOKEN` by default, so it no longer needs to be passed manually.
See #57572.
git-svn-id: https://develop.svn.wordpress.org/trunk@55487 602fd350-edb4-49c9-b593-d223f7449a82
This is a follow up to r55459, which fixes the hostname used for logging performance metrics by adding the correct www subdomain.
See #57687.
git-svn-id: https://develop.svn.wordpress.org/trunk@55479 602fd350-edb4-49c9-b593-d223f7449a82
This adds a new GitHub Action workflow that measures a set of performance metrics on every commit, so we can track changes in the performance of WordPress over time and more easily identify changes that are responsible for significant performance improvements or regressions during development cycles.
The workflow measures the homepage of a classic theme (Twenty Twenty-One) and a block theme (Twenty Twenty-Three) set up with demo content from the Theme Test Data project. Using the e2e testing framework, it makes 20 requests and records the median value of the following Server Timing metrics, generated by an mu-plugin installed as part of this workflow:
- Total server response time
- Server time before templates are loaded
- Server time during template rendering
In addition to measuring the performance metrics of the current commit, it also records performance metrics of a consistent version of WordPress (6.1.1) to be used as a baseline measurement in order to remove variance caused by the GitHub workers themselves from our reporting.
The measurements are collected and displayed at https://www.codevitals.run/project/wordpress.
Props adamsilverstein, mukesh27, flixos90, youknowriad, oandregal, desrosj, costdev, swissspidy.
Fixes#57687.
git-svn-id: https://develop.svn.wordpress.org/trunk@55459 602fd350-edb4-49c9-b593-d223f7449a82
This updates the following third-party GitHub Actions to their latest versions.
- `actions/cache`
- `actions/checkout`
- `actions/upload-artifact`
- `actions/setup-node`
- `actions/setup-php`
- `actions/github-script`
See #57572.
git-svn-id: https://develop.svn.wordpress.org/trunk@55152 602fd350-edb4-49c9-b593-d223f7449a82
We're gonna teach 'em how to say goodbye (teach 'em how to say goodbye)
Teach 'em how
To say goodbye
To say goodbye (say goodbye)
Say goodbye (say goodbye)
One last time
Fixes#57228.
git-svn-id: https://develop.svn.wordpress.org/trunk@54995 602fd350-edb4-49c9-b593-d223f7449a82
This improves the speed of the PHPCS scans between workflow runs in GitHub Action by creating a scan cache file for each unique set of arguments passed to `phpcs`.
Props jrf.
Fixes#57148. See #53841.
git-svn-id: https://develop.svn.wordpress.org/trunk@54921 602fd350-edb4-49c9-b593-d223f7449a82
To improve how Composer dependencies are installed and managed within GitHub Actions, the `ramsey/composer-install` third-party action is now used consistently throughout all workflows.
Previously, some workflows manually ran `composer` commands while others already used `ramsey/composer-install`.
The `ramsey/composer-install` action manages caching dependencies across workflow runs internally, which is something that was manually handled before this change.
Props jrf, desrosj.
Fixes#53841.
git-svn-id: https://develop.svn.wordpress.org/trunk@54856 602fd350-edb4-49c9-b593-d223f7449a82
