d' (d prime) stems from signal detection theory. You'll find the basic tenets of SDT as well as the assumptions underlying d' covered in any introductory texts on SDT (I personally like "Elementary Signal Detection Theory" by Wickens). d' is generally applicable to 2-choice tasks and is a measure of *sensitivity*, i.e., the ability of the receiver (participant) to distinguish between signal and noise.
Mathematically, d' reflects the *distance* between the mean of the *signal* distribution and the mean of the *noise* distribution in terms of standard deviations. Applied to a go/no-go paradigm like the GNAT, "go"-trials are signal-trials whereas "no-go" trials are noise-trials.
The theoretical maximum of d' is +infinity, which would indicate *perfect* performance / perfect ability to distinguish between signal and noise. Again, applied to the GNAT, this would mean a participant responding *correctly* in every trial (i.e., responding in all "go"-trials, refraining from responding in all "no-go" trials).
A d' value of zero would indicate chance performance, i.e. an inability to properly distinguish between signal and noise.
The theoretical minimum of d' is -infinity, and as you correctly state, negative d' values indicate *below chance* performance due to response confusion, misunderstanding the task or other (unknown) factors. The fact that your participants achieve positive d' values after having been re-briefed lends support to the confusion / misunderstanding hypothesis. In addition note that d' is an estimator -- *small* negative values can result due to sampling error and may be indicative of a "true" d' near zero (see above).
For a more rigorous treatment of SDT measures, including the assumptions underlying d', see e.g. Stanislaw & Todorov (1999;
https://www.researchgate.net/file.PostFileLoader.html?id=536ce28fd5a3f2b41b8b46c5&assetKey=AS%3A273596423835649%401442241878760 ).