The language option now has a clearly defined fallback chain (#5602)

* Made the test suite for translations more complete

There were previously tests for translations within the test suite,
but they only covered a select few problem cases that had previously
been fixed. They did not cover the majority of cases, which makes
changes to how the translation mechanism for Select2 works a bit
more challenging.

So this adds tests for the majority of edge cases around translations,
including how one would expect the fallback chains to work and also
around how defaults interact with the language options. This should
not be considered an exhaustive list of all of the edge cases, but
it should be good enough to refactor the internals and not have to
worry as much.

The one change of note to this test file is that we are now properly
resetting the defaults in between tests. This should fix any issues
that we may have seen where the defaults were not being reset, and
thus tests were not properly isolated and would start to interfere
with each other. This required pulling the module definition down
below the imports, since we need to reference the defaults within
the module definition.

Many of these tests will fail because the translation system is
broken in many small, unrealized ways. The next few commits should
make these pass and fix the issues that we are seeing.

* Consistently resolve the language option

This fixes an issue that we have had for a while where we did not
have a way to consistently take the `language` option from a string,
array, object, or whatever else is specified and resolve it into
`Translation`-compatible objects or strings. Now there is a new
internal `_resolveLanguage` function which is able to take any of
the supported ways of specifying a language and resolve it down into
the supported language chain.

This now means that we can properly resolve the following cases,
and we can do it in a consistent manner.

* When the language is specified as just a string (for example: "en")
* When the language is specified as a string containing a region
  (for example: "en-US")
* When the langugae chian is specified as a list of strings (for
  example: ["es", "en"])
* When the language is specifid as an object containing messages
  (for example, when a user overrides only a subset of messages)
* When the language is specified as a list of strings and objects
  (for example, when a user wants to use a language other than
  English and also wants to ovverride some default messages)
* When the language is not specified at all (the most common case)
* When the language is specified as an empty object (an edge case
  that allows us to skip processing it)

This allows us to consistently produce the language fallback chain
based on the given `language` option, something which we could not
actually do before because we didn't have a consistent chain. This also
means that now the `language` option will consistently be an array
after going through this process, instead of being any number of
types before.

The translation generation currently does not support having objects
and strings mixed as a part of the fallback chain, despite that being
how the default chain has always worked, and as such there are still
failing tests around this.

* Move English to always be at the end of the language chain

This was technically true in most cases in the past, because if a
language chain was manually specified then it would have English
injected into the end of it anyway. This is needed because not all
translations are complete, but we know the English one is, and
Select2 relies on the translation that it uses being complete.

This will result in cases where a user specifies a language but still
receives English translation for some things, which is what users
have historically seen when using partial translations anyway. This
just ensures that there will always be a complete translation that
is being used, so they won't get unexpected errors and will instead
get unexpected English translations.

* Filter out repeated languages in fallback chain

This is mostly being done for performance reasons, since Select2
will not behave any differently when there are duplicates, but it
makes things cleaner when you ask for the fallback chain and it
only contains unique values.

This cannot distinguish between languages specified by name (string)
and languages specified by the contents of their language file
(such as the default, English), but this should generally not be
an issue.

* Convert the language chain into a finalized translation

This extracts the logic for converting parts of the language chain
into the finalized `Translation` objects out into its own method,
with some small fixes for edge cases.

This can now properly convert a language chain containing both
strings and objects into a translation object that contains them
both.

We no longer need to special case the `language` option being an
array since we know that it will be an array once the language
resolution process is completed.

* Switch default translation to be empty

This should have no external effects, but it fixes an interesting
bug where resetting the defaults would not always reset custom
translations. This was because it was possible to modify the
included English translation when you were setting a default for the
language option.

This should not cause any issues because the English translation is
now appended to the end of the language chain when the defaults
are applied, which means that English will continue to exist as the
final fallback.

* Inherited `lang` attribute should be below the default in the chain

It was pointed out in #5468 that the `lang` attribute, when inherited
from a parent element, was above the option set in the global default
within the inheritance chain. While this makes sense because of how
we inherit other properties, it does not make sense for the `lang`
attribute.

The inheritance chain for the `language` option has been adjusted
to be the following:

1. The `lang` attribute on the original `<select>` element
2. The `data-language` attribute on the original `<select>` element
   (because of how `data-*` attribute resolution affects options)
3. The `language` option specified when initiailizing Select2
4. The `language` Select2 default
5. The `lang` attribute on a parent of the `<select>` element

While this is a breaking change, we believe that this change will
have minimal to no impact on real-world usage of Select2, because
of how the `lang` attribute is generally used within documents.
We believe this will now make setting the default language through
JavaScript easier and more reliable, bringing it in line with how
we recommend it is done within the documentation.

This was implemented through a new method `Defaults.applyFromElement`
instead of within the old `Options.fromElement` method because it
relies on the global defaults object. While we could have reached
in to the internals in order to apply it appropriately, it made
more sense to handle the proper resolution through a single
consistent place.

This was not implemented in the `Defaults.apply` method because that
method is not typically passed in a reference to the original
`<select>` element that it is applying the options for.

Closes #5468

* Properly resolve language chains with dictionaries

It is possible for a language chain to be specified that includes
both a dictionary and the string translation as a part of the same
chain, but we would previously throw an error because we assumed
that the list could only contain strings and that dictionaries
would never be included in lists.

This fixes the issue so language region normalization only occurs
when a string is specified in the language chain, which is what we
were previously assuming was the case but was not actually.

This also now resolves the entire language option during the
`Defaults.apply` method. This should be a no-op except for internal
tests, because the `Defaults.applyFromElement` method should almost
always be called in real-world scenarios.

src/js/select2/defaults.js

@@ -232,66 +232,29 @@ define([
       );
     }
 
-    if (typeof options.language === 'string') {
+    // If the defaults were not previously applied from an element, it is
-      // Check if the language is specified with a region
+    // possible for the language option to have not been resolved
-      if (options.language.indexOf('-') > 0) {
+    options.language = this._resolveLanguage(options.language);
-        // Extract the region information if it is included
-        var languageParts = options.language.split('-');
-        var baseLanguage = languageParts[0];
 
-        options.language = [options.language, baseLanguage];
+    // Always fall back to English since it will always be complete
-      } else {
-        options.language = [options.language];
-      }
-    }
-
-    if ($.isArray(options.language)) {
-      var languages = new Translation();
     options.language.push('en');
 
-      var languageNames = options.language;
+    var uniqueLanguages = [];
 
-      for (var l = 0; l < languageNames.length; l++) {
+    for (var l = 0; l < options.language.length; l++) {
-        var name = languageNames[l];
+      var language = options.language[l];
-        var language = {};
 
-        try {
+      if (uniqueLanguages.indexOf(language) === -1) {
-          // Try to load it with the original name
+        uniqueLanguages.push(language);
-          language = Translation.loadPath(name);
+      }
-        } catch (e) {
+    }
-          try {
+
-            // If we couldn't load it, check if it wasn't the full path
+    options.language = uniqueLanguages;
-            name = this.defaults.amdLanguageBase + name;
+
-            language = Translation.loadPath(name);
+    options.translations = this._processTranslations(
-          } catch (ex) {
+      options.language,
-            // The translation could not be loaded at all. Sometimes this is
+      options.debug
-            // because of a configuration problem, other times this can be
-            // because of how Select2 helps load all possible translation files.
-            if (options.debug && window.console && console.warn) {
-              console.warn(
-                'Select2: The language file for "' + name + '" could not be ' +
-                'automatically loaded. A fallback will be used instead.'
     );
-            }
-
-            continue;
-          }
-        }
-
-        languages.extend(language);
-      }
-
-      options.translations = languages;
-    } else {
-      var baseTranslation = Translation.loadPath(
-        this.defaults.amdLanguageBase + 'en'
-      );
-      var customTranslation = new Translation(options.language);
-
-      customTranslation.extend(baseTranslation);
-
-      options.translations = customTranslation;
-    }
 
     return options;
   };
@@ -358,7 +321,7 @@ define([
       debug: false,
       dropdownAutoWidth: false,
       escapeMarkup: Utils.escapeMarkup,
-      language: EnglishTranslation,
+      language: {},
       matcher: matcher,
       minimumInputLength: 0,
       maximumInputLength: 0,
@@ -380,6 +343,103 @@ define([
     };
   };
 
+  Defaults.prototype.applyFromElement = function (options, $element) {
+    var optionLanguage = options.language;
+    var defaultLanguage = this.defaults.language;
+    var elementLanguage = $element.prop('lang');
+    var parentLanguage = $element.closest('[lang]').prop('lang');
+
+    var languages = Array.prototype.concat.call(
+      this._resolveLanguage(elementLanguage),
+      this._resolveLanguage(optionLanguage),
+      this._resolveLanguage(defaultLanguage),
+      this._resolveLanguage(parentLanguage)
+    );
+
+    options.language = languages;
+
+    return options;
+  };
+
+  Defaults.prototype._resolveLanguage = function (language) {
+    if (!language) {
+      return [];
+    }
+
+    if ($.isEmptyObject(language)) {
+      return [];
+    }
+
+    if ($.isPlainObject(language)) {
+      return [language];
+    }
+
+    var languages;
+
+    if (!$.isArray(language)) {
+      languages = [language];
+    } else {
+      languages = language;
+    }
+
+    var resolvedLanguages = [];
+
+    for (var l = 0; l < languages.length; l++) {
+      resolvedLanguages.push(languages[l]);
+
+      if (typeof languages[l] === 'string' && languages[l].indexOf('-') > 0) {
+        // Extract the region information if it is included
+        var languageParts = languages[l].split('-');
+        var baseLanguage = languageParts[0];
+
+        resolvedLanguages.push(baseLanguage);
+      }
+    }
+
+    return resolvedLanguages;
+  };
+
+  Defaults.prototype._processTranslations = function (languages, debug) {
+    var translations = new Translation();
+
+    for (var l = 0; l < languages.length; l++) {
+      var languageData = new Translation();
+
+      var language = languages[l];
+
+      if (typeof language === 'string') {
+        try {
+          // Try to load it with the original name
+          languageData = Translation.loadPath(language);
+        } catch (e) {
+          try {
+            // If we couldn't load it, check if it wasn't the full path
+            language = this.defaults.amdLanguageBase + language;
+            languageData = Translation.loadPath(language);
+          } catch (ex) {
+            // The translation could not be loaded at all. Sometimes this is
+            // because of a configuration problem, other times this can be
+            // because of how Select2 helps load all possible translation files
+            if (debug && window.console && console.warn) {
+              console.warn(
+                'Select2: The language file for "' + language + '" could ' +
+                'not be automatically loaded. A fallback will be used instead.'
+              );
+            }
+          }
+        }
+      } else if ($.isPlainObject(language)) {
+        languageData = new Translation(language);
+      } else {
+        languageData = language;
+      }
+
+      translations.extend(languageData);
+    }
+
+    return translations;
+  };
+
   Defaults.prototype.set = function (key, value) {
     var camelKey = $.camelCase(key);
 

src/js/select2/options.js

@@ -11,6 +11,10 @@ define([
       this.fromElement($element);
     }
 
+    if ($element != null) {
+      this.options = Defaults.applyFromElement(this.options, $element);
+    }
+
     this.options = Defaults.apply(this.options);
 
     if ($element && $element.is('input')) {
@@ -34,14 +38,6 @@ define([
       this.options.disabled = $e.prop('disabled');
     }
 
-    if (this.options.language == null) {
-      if ($e.prop('lang')) {
-        this.options.language = $e.prop('lang').toLowerCase();
-      } else if ($e.closest('[lang]').prop('lang')) {
-        this.options.language = $e.closest('[lang]').prop('lang');
-      }
-    }
-
     if (this.options.dir == null) {
       if ($e.prop('dir')) {
         this.options.dir = $e.prop('dir');

tests/options/translation-tests.js

@@ -1,16 +1,73 @@
-module('Options - Translations');
-
 var $ = require('jquery');
 var Options = require('select2/options');
+var Defaults = require('select2/defaults');
+
+module('Options - Translations', {
+  beforeEach: function () {
+    Defaults.reset();
+  },
+  afterEach: function () {
+    Defaults.reset();
+  }
+});
+
+test('partial dictonaries are reset when default reset', function (assert) {
+  Defaults.set('language', {
+    test: 'testing'
+  });
+
+  Defaults.reset();
+
+  assert.ok(
+    !Defaults.defaults.language.test,
+    'The partial dictionary should have been reset'
+  );
+});
+
+test('default language chain is English', function (assert) {
+  var $element = $('<select></select>');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['en']
+  );
+});
+
+test(
+  'default translation includes all of the required messages',
+  function (assert) {
+    var $element = $('<select></select>');
+
+    var options = new Options({}, $element);
+
+    assert.deepEqual(
+      Object.keys(options.get('translations').all()),
+      [
+        'errorLoading',
+        'inputTooLong',
+        'inputTooShort',
+        'loadingMore',
+        'maximumSelected',
+        'noResults',
+        'searching',
+        'removeAllItems'
+      ]
+    );
+  }
+);
 
 test('partial dictionaries can be passed', function (assert) {
+  var $element = $('<select></select>');
+
   var options = new Options({
     language: {
       searching: function () {
         return 'Something';
       }
     }
-  });
+  }, $element);
 
   var translations = options.get('translations');
 
@@ -26,3 +83,206 @@ test('partial dictionaries can be passed', function (assert) {
     'You can still get English translations for keys not passed in'
   );
 });
+
+test('partial dictionaries can be combined with defaults', function (assert) {
+  var $element = $('<select></select>');
+
+  Defaults.set('language', {
+    test: function () {
+      return 'Testing';
+    }
+  });
+
+  var options = new Options({
+    language: {
+      searching: function () {
+        return 'Something';
+      }
+    }
+  }, $element);
+
+  var translations = options.get('translations');
+
+  assert.equal(
+    translations.get('searching')(),
+    'Something',
+    'The partial dictionary still overrides translations'
+  );
+
+  assert.equal(
+    translations.get('test')(),
+    'Testing',
+    'The defaults were included in the fallback chain'
+  );
+
+  assert.equal(
+    translations.get('noResults')(),
+    'No results found',
+    'You can still get English translations for keys not passed in'
+  );
+});
+
+test('partial dictionaries can used in fallback chains', function (assert) {
+  var $element = $('<select></select>');
+
+  var options = new Options({
+    language: [
+      {
+        searching: function () {
+          return 'Something';
+        }
+      },
+      {
+        test: function () {
+          return 'Testing';
+        }
+      }
+    ]
+  }, $element);
+
+  var translations = options.get('translations');
+
+  assert.equal(
+    translations.get('searching')(),
+    'Something',
+    'The partial dictionary still overrides translations'
+  );
+
+  assert.equal(
+    translations.get('test')(),
+    'Testing',
+    'The defaults were included in the fallback chain'
+  );
+
+  assert.equal(
+    translations.get('noResults')(),
+    'No results found',
+    'You can still get English translations for keys not passed in'
+  );
+});
+
+test('language can be set via the options', function (assert) {
+  var $element = $('<select></select>');
+
+  var options = new Options({
+    language: 'es'
+  }, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['es', 'en']
+  );
+});
+
+test('multi-part language is broken out', function (assert) {
+  var $element = $('<select></select>');
+
+  var options = new Options({
+    language: 'pt-BR'
+  }, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['pt-BR', 'pt', 'en']
+  );
+});
+
+test('default language can be set', function (assert) {
+  var $element = $('<select></select>');
+
+  Defaults.set('language', 'es');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['es', 'en']
+  );
+});
+
+test('lanugage set via options adds to default chain', function (assert) {
+  var $element = $('<select></select>');
+
+  Defaults.set('language', 'es');
+
+  var options = new Options({
+    language: 'it'
+  }, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['it', 'es', 'en']
+  );
+});
+
+test('default language chain can be set', function (assert) {
+  var $element = $('<select></select>');
+
+  Defaults.set('language', ['es', 'it', 'en']);
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['es', 'it', 'en']
+  );
+});
+
+test('language can be set by lang attr', function (assert) {
+  var $element = $('<select lang="es"></select>');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['es', 'en']
+  );
+});
+
+test('language can be inherited by lang attr', function (assert) {
+  var $element = $('<div lang="es"><select></select></div>').find('select');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['es', 'en']
+  );
+});
+
+test('multi-part language can be inherited by lang attr', function (assert) {
+  var $element = $('<div lang="pt-BR"><select></select></div>').find('select');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['pt-BR', 'pt', 'en']
+  );
+});
+
+test('lang attr overrides default language', function (assert) {
+  var $element = $('<select lang="it"></select>');
+
+  Defaults.set('language', 'es');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['it', 'es', 'en']
+  );
+});
+
+test('default language overrides inherited lang attr', function (assert) {
+  var $element = $('<div lang="it"><select></select></div>').find('select');
+
+  Defaults.set('language', 'es');
+
+  var options = new Options({}, $element);
+
+  assert.deepEqual(
+    options.get('language'),
+    ['es', 'it', 'en']
+  );
+});