When do I use a document schema, and when do I use an extension schema?
The GJXDM and NIEM schemas (and subsets thereof) contain global element declarations. If there are multiple global element declarations in a schema, a validating parser is not capable of determining which of those global elements is intended to be the root of a valid instance. (Some parsers consider this an error, others a warning; the XML Schema specification is ambiguous on the proper parser behavior.)
For this reason, it is a best practice for every IEPD to include a document schema that identifies the single element that is the root of a valid instance. The document schema defines an IEPD-specific namespace that contains only a single element declaration. The element's type should be in either the GJXDM or NIEM namespace or an IEPD-specific extension namespace.
So, in summary answer to the question: Always use a document schema when it is important to you to identify the root element of a valid instance. Since this is almost always desired, it is a best practice always to define a document schema.
An extension schema defines an IEPD-specific namespace that contains types and elements that are particular to that IEPD. Types in an extension schema should extend types in GJXDM or NIEM; elements in an extension schema should be either of types in the extension schema, or types in GJXDM or NIEM.
It is typical for an IEPD to have at least a few extensions. Extension schemas play a valuable role in the concept of conformance. While it is important for IEPDs to leverage types and elements in GJXDM and NIEM if they fit the semantics of the exchange, it is equally important not to use a type or element from them if its definition does not fit the semantics. It is very important not to use a GJXDM or NIEM element solely on the basis of the name "sounding like" what is intended in the exchange semantics; the definition must be consistent with those semantics as well.
So, in summary answer to the question: Use an extension schema whenever the exchange involves semantics that do not exist in GJXDM or NIEM.