Formatting: Normalize to Unicode NFC encoding before converting accent characters in remove_accents().

This changeset adds Unicode sequence normalization from NFD to NFC, via the `normalizer_normalize()` PHP function which is available with the recommended `intl` PHP extension.

This fixes an issue where NFD characters were not properly sanitized. It also provides a unit test for NFD sequences (alternate Unicode representations of the same characters).

Props NumidWasNotAvailable, targz, nacin, nunomorgadinho, p_enrique, gitlost, SergeyBiryukov, markoheijnen, mikeschroder, ocean90, pento, helen, rodrigosevero, zodiac1978, ironprogrammer, audrasjb, azaozz, laboiteare, nuryko, virgar, dxd5001, onnimonni, johnbillion.
Fixes #24661, #47763, #35951.
See #30130, #52654.


git-svn-id: https://develop.svn.wordpress.org/trunk@53754 602fd350-edb4-49c9-b593-d223f7449a82

tests/phpunit/tests/formatting/removeAccents.php

@@ -11,6 +11,24 @@ class Tests_Formatting_RemoveAccents extends WP_UnitTestCase {
 		$this->assertSame( 'abcdefghijkl', remove_accents( 'abcdefghijkl' ) );
 	}
 
+	/**
+	 * @ticket 24661
+	 *
+	 * Tests Unicode sequence normalization from NFD (Normalization Form Decomposed)
+	 * to NFC (Normalization Form [Pre]Composed), the encoding used in `remove_accents()`.
+	 *
+	 * For more information on Unicode normalization, see
+	 * https://unicode.org/faq/normalization.html.
+	 *
+	 * @requires extension intl
+	 */
+	public function test_remove_accents_latin1_supplement_nfd_encoding() {
+		$input  = 'ªºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ';
+		$output = 'aoAAAAAAAECEEEEIIIIDNOOOOOOUUUUYTHsaaaaaaaeceeeeiiiidnoooooouuuuythy';
+
+		$this->assertSame( $output, remove_accents( $input ), 'remove_accents replaces Latin-1 Supplement with NFD encoding' );
+	}
+
 	/**
 	 * @ticket 9591
 	 */