I tried hardly to associate a custom player for some specific contents on my Kodi installation. While defining an external player and some custom rules to associate it to some contents is definitely possible, I was not able to execute a program that can take control of the graphic screen. See this paragraph about executing a graphical external program.

Neverthless, here are the notes about defining the external player.

You have to create/customize the file userdata/playercorefactory.xml, something like this:

<playercorefactory>
  <players>
    <player name="Photo-Reframe" type="ExternalPlayer" audio="false" video="true">
      <filename>/usr/local/bin/external-player.sh</filename>
      <args>--play --fullscreen --read-only "{0}"</args>
      <hidexbmc>true</hidexbmc>
      <hideconsole>false</hideconsole>
      <warpcursor>none</warpcursor>
    </player>
  </players>
  <rules action="prepend">
    <rule filetypes="m3u" filename=".*playlist.*" player="Photo-Reframe"/>
  </rules>
</playercorefactory>

Into the kodi.log you can read that the player was actually created, but you have to enable the DEBUG logging (Settings ⇒ System ⇒ Logging ⇒ Enable debug logging), otherwise the WARNING message is rather misleading:

WARNING <general>: CPlayerCoreFactory::GetPlayer(Photo-Reframe): no such player: Photo-Reframe
  DEBUG <general>: CPlayerCoreConfig::<ctor>: created player Photo-Reframe

Once the external player is loaded, you can select a video and open the context menu; under the Play using… item you will find your external player. Notice that the Play using… context menu is available only into the standard video interface, it does not appear e.g. from the Movie ⇒ Recently added movies interface.

The external player can be any executable (even a shell script), the process will receive the arguments defined by the <args> element ad some environment variables, e.g.:

KODI_BIN_HOME='/usr/lib/arm-linux-gnueabihf/kodi'
KODI_HOME='/usr/share/kodi'
LIRC_SOCKET_PATH='/var/run/lirc/lircd'

With the appropriate values into the <rules> and <rule> elements of playercorefactory.xml, you can make the external player the default one for some specific media. Using the context menu ⇒ Play using… item, you can always confirm that the specified rules were applied because the default player is highlighted with the (Default) label appended.

WARNING: The filename attribute is a regexp applied to the entire item path, so if you intend to match only the file name part (and not the path) you must start the expression with .*, which means ““any character repeated zero or more times””.

When writing a custom video plugin add-on, I tried to apply a video external player to files that are actually playlists of images, i.e. text files with an m3u extension. From the context menu it was clear that they were now recognized as video files because the Play using… item was not present and the default player was not the video player, despite the filename and filetypes attributes were properly specified into the playercorefactory.xml file. Use the MimeType property?