The complexity of deciding characteristic formulae modulo nested simulation

This paper studies the complexity of determining whether a formula in the modal logics characterizing the nested-simulation semantics is characteristic for some process, which is equivalent to determining whether the formula is satisfiable and prime. The main results are that the problem of determining whether a formula is prime in the modal logic characterizing the 2-nested-simulation preorder is CoNP-complete and is PSPACE-complete in the case of the n-nested-simulation preorder, when n>= 3. This establishes that deciding characteristic formulae for the n-nested simulation semantics, n>= 3, is PSPACE-complete. In the case of the 2-nested simulation semantics, that problem lies in the complexity class DP, which consists of languages that can be expressed as the intersection of one language in NP and of one in CoNP.