This key combination is a toggle that “Displays or hides the formulas instead of the values in all cells.” (Search for the back-tick on that page to see the instructions/explanation.)
This is a known bug: fdo#103388. At least since January it has been registered as an issue for OSX. (And I can confirm!) [ Update - As noted by @vsfoote, this has been fixed, and will be available when you get v. 6.0.3+ for OSX. I’m still on 6.0.1, though.]
There is a workaround, but it is very much USE AT YOUR OWN RISK 
-
Open LibO Preferences (⌘,)
-
Open the first category, LibreOffice > Advanced,
-
click on Open Expert Configuration.
-
in the “search” bar in the new pop-up, type: “toggleFormula” and hit ENTER. This will bring up a function that includes QUOTELEFT_MOD1:
Double-click that line, and clear the value .uno:ToggleFormula from the box that pops up.
-
Click OK (for each of the three pop-up windows).
Now ⌘` will work on the system level, and you’ll be able to use that key combination to switch between Calc windows.
It is a simple matter to reverse this operation, of course, but you now need to search on QUOTELEFT_MOD1 to find that function.