For simple custom detectors, have you looked into using the appid_detector_builder.sh script under the tools/ folder? That should give you a good start.
I am doing a college work on OpenAppID (Snort 188.8.131.52/Ubuntu16). I'm having trouble creating a detector for a third-party application. I read OpenDetectorDeveloperGuide3.0n (https://www.snort.org/downloads/openappid/6328), but it was not clear how to customize a detector. Someone with knowledge about the subject?