Why am I getting grid when I want 5-stave perc. kit?

See images, I believe I have selected everything correctly.

  1. Library > Layout Options.
  2. Ensure that the correct layout is selected in the right side of the dialog.
  3. Players > Percussion.
  4. See appropriate kit to give line staff.
  5. Apply and Close.

The Percussion Kit Editor (shown in your first screenshot) is where you setup your Percussion Kits. The tabs at the top determine how your kit appears within each of the different representations, but does not determine which representation appears in your layout. The fact that the representation settings are in Layout Options make it possible to represent a percussion kit differently in different layouts; e.g. it can be useful to slim things down to a five-line staff in the score while showing separate instruments in the part (particularly if different people are playing individual instruments from a percussion score).