userdata/playercorefactory.xml

<playercorefactory>
  <players>
    <player name="Photo-Reframe" type="ExternalPlayer" audio="false" video="true">
      <filename>/usr/local/bin/external-player.sh</filename>
      <args>--play --fullscreen --read-only "{0}"</args>
      <hidexbmc>true</hidexbmc>
      <hideconsole>false</hideconsole>
      <warpcursor>none</warpcursor>
    </player>
  </players>
  <rules action="prepend">
    <rule filetypes="m3u" filename=".*playlist.*" player="Photo-Reframe"/>
  </rules>
</playercorefactory>

Into the kodi.log you can read that the player was actually created, but you have to enable the DEBUG logging (Settings ⇒ System ⇒ Logging ⇒ Enable debug logging), otherwise the WARNING message is rather misleading:

WARNING <general>: CPlayerCoreFactory::GetPlayer(Photo-Reframe): no such player: Photo-Reframe
  DEBUG <general>: CPlayerCoreConfig::<ctor>: created player Photo-Reframe

Once the external player is loaded, you can select a video and open the context menu; under the Play using… item you will find your external player. Notice that the Play using… context menu is available only into the standard video interface, it does not appear e.g. from the Movie ⇒ Recently added movies interface.

The external player can be any executable (even a shell script), the process will receive the arguments defined by the <args> element ad some environment variables, e.g.:

KODI_BIN_HOME='/usr/lib/arm-linux-gnueabihf/kodi'
KODI_HOME='/usr/share/kodi'
LIRC_SOCKET_PATH='/var/run/lirc/lircd'

With the appropriate values into the <rules> and <rule> elements of playercorefactory.xml, you can make the external player the default one for some specific media. Using the context menu ⇒ Play using… item, you can always confirm that the specified rules were applied because the default player is highlighted with the (Default) label appended.

WARNING: The filename attribute is a regexp applied to the entire item path, so if you intend to match only the file name part (and not the path) you must start the expression with .*, which means ““any character repeated zero or more times””.