[Summary view] [Print] [Text view]

   1  @title Managing Storage Adjustments
   2  @group config
   3  
   4  Explains how to apply storage adjustments to the MySQL schemata.
   5  
   6  Overview
   7  ========
   8  
   9  Phabricator uses a workflow called //storage adjustment// to make some minor
  10  kinds of changes to the MySQL schema. This workflow compliments the //storage
  11  upgrade// workflow, which makes major changes.
  12  
  13  You can perform storage adjustment by running:
  14  
  15    phabricator/ $ ./bin/storage adjust
  16  
  17  This document describes what adjustments are, how they relate to storage
  18  upgrades, how to perform them, and how to troubleshoot issues with storage
  19  adjustment.
  20  
  21  
  22  Understanding Adjustments
  23  ===================
  24  
  25  Storage adjustments make minor changes to the Phabricator MySQL schemata to
  26  improve consistency, unicode handling, and performance. Changes covered by
  27  adjustment include:
  28  
  29    - Character set and collation settings for columns, tables, and databases.
  30    - Setting and removing "Auto Increment" on columns.
  31    - Adding, removing, renaming and adjusting keys.
  32  
  33  Adjustment does not make major changes to the schemata, like creating or
  34  removing columns or tables or migrating data. (Major changes are performed by
  35  the upgrade workflow.)
  36  
  37  Adjustments are separate from upgrades primarily because adjustments depend on
  38  the MySQL version, while upgrades do not. If you update MySQL, better collations
  39  may become available, and the adjustment workflow will convert your schemata to
  40  use them.
  41  
  42  All changes covered by adjustment are minor, and technically optional. However,
  43  you are strongly encouraged to apply outstanding adjustments: if you do not,
  44  you may encounter issues storing or sorting some unicode data, and may suffer
  45  poor performance on some queries.
  46  
  47  
  48  Reviewing Outstanding Adjustments
  49  =================================
  50  
  51  There are two ways to review outstanding adjustments: you can use the web UI,
  52  or you can use the CLI.
  53  
  54  To access the web UI, navigate to {nav Config > Database Status} or
  55  {nav Config > Database Issues}. The //Database Status// panel provides a general
  56  overview of all schemata. The //Database Issues// panel shows outstanding
  57  issues.
  58  
  59  These interfaces report //Errors//, which are serious issues that can not be
  60  resolved through adjustment, and //Warnings//, which are minor issues that the
  61  adjustment workflow can resolve.
  62  
  63  You can also review adjustments from the CLI, by running:
  64  
  65    phabricator/ $ ./bin/storage adjust
  66  
  67  Before you're prompted to actually apply adjustments, you'll be given a list of
  68  available adjustments. You can then make a choice to apply them.
  69  
  70  
  71  Performing Adjustments
  72  ======================
  73  
  74  To perform adjustments, run the `adjust` workflow:
  75  
  76    phabricator/ $ ./bin/storage adjust
  77  
  78  For details about flags, use:
  79  
  80    phabricator/ $ ./bin/storage help adjust
  81  
  82  You do not normally need to run this workflow manually: it will be run
  83  automatically after you run the `upgrade` workflow.
  84  
  85  
  86  History and Rationale
  87  =====================
  88  
  89  The primary motivation for the adjustment workflow is MySQL's handling of
  90  unicode character sets. Before MySQL 5.5, MySQL supports a character set called
  91  `utf8`. However, this character set can not store 4-byte unicode characters
  92  (including emoji). Inserting 4-byte characters into a `utf8` column truncates
  93  the data.
  94  
  95  With MySQL 5.5, a new `utf8mb4` character set was introduced. This character
  96  set can safely store 4-byte unicode characters.
  97  
  98  The adjustment workflow allows us to alter the schema to primarily use
  99  `binary` character sets on older MySQL, and primarily use `utf8mb4` character
 100  sets on newer MySQL. The net effect is that Phabricator works consistently and
 101  can store 4-byte unicode characters regardless of the MySQL version. Under
 102  newer MySQL, we can also take advantage of the better collation rules the
 103  `utf8mb4` character set offers.
 104  
 105  The adjustment workflow was introduced in November 2014. If your install
 106  predates its introduction, your first adjustment may take a long time (we must
 107  convert all of the data out of `utf8` and into the appropriate character set).
 108  If your install was set up after November 2014, adjustments should generally
 109  be very minor and complete quickly, unless you perform a major MySQL update and
 110  make new character sets available.
 111  
 112  If you plan to update MySQL from an older version to 5.5 or newer, it is
 113  advisable to update first, then run the adjustment workflow. If you adjust
 114  first, you'll need to adjust again after updating, so you'll end up spending
 115  twice as much time performing schemata adjustments.
 116  
 117  
 118  Troubleshooting
 119  ===============
 120  
 121  When you apply adjustments, some adjustments may fail. The two most common
 122  errors you may encounter are:
 123  
 124    - **#1406 Data Too Long**: Usually this is caused by a very long object name
 125      (like a task title) which contains multibyte unicode characters. When the
 126      column type is converted to `binary`, only the first part of the title still
 127      fits in the column. Depending on what is failing, you may be able to find
 128      the relevant object in the web UI and retitle it so the adjustment succeeds.
 129      Alternatively, you can use `--unsafe` to force the adjustment to truncate
 130      the title. This will destroy some data, but usually the data is not
 131      important (just the end of very long titles).
 132    - **#1366 Incorrect String Value**: This can occur when converting invalid
 133      or truncated multibyte unicode characters to a unicode character set.
 134      In both cases, the old value can not be represented under the new character
 135      set. You may be able to identify the object and edit it to allow the
 136      adjustment to proceed, or you can use the `--unsafe` flag to truncate the
 137      data at the invalid character. Usually, the truncated data is not important.
 138  
 139  As with most commands, you can add the `--trace` flag to get more details about
 140  what `bin/storage adjust` is doing. This may help you diagnose or understand any
 141  issues you encounter, and this data is useful if you file reports in the
 142  upstream.
 143  
 144  In general, adjustments are not critical. If you run into issues applying
 145  adjustments, it is safe to file a task in the upstream describing the problem
 146  you've encountered and continue using Phabricator normally until the issue can
 147  be resolved.