Incorrect handling of OPTIONAL when in query with textmatch functional predicate

(In [2214]) re #316 reviewer: lee

Add unit test to replicate defect.

This issue not occur when the OPTIONAL is below the mandatory triple patterns:

SELECT DISTINCT ?value
WHERE {
  ?value ?p ?txt .
  ?txt <http://openanzo.org/predicates/textmatch> 'john' .
  OPTIONAL { ?value <http://xmlns.com/foaf/0.1/givenname> ?givenname } .
}

correctly returns: ?value -> :s for the dataset containing graph2.

But if the OPTIONAL is above, as in the initial report of this defect, the Glitter AST rewriter do not put the mandatory triple patterns in the OPTIONAL's nodes 'must' child, node, instead the AST looks like:

Group(
  Optional(
    null,
    Group(BGP(TriplePatternNode(?value <http://xmlns.com/foaf/0.1/givenname> ?obj)))),
  BGP(
    TriplePatternNode(?txt <http://openanzo.org/predicates/textmatch> "union"),
    TriplePatternNode(?value ?p ?txt))
)

Where null is the 'must' child of the optional and the mandatory triple patterns are in a separate BGP that will be joined with the results of the optional.

From the SPARQL specification it is unclear to me if this is correct or not. In the spec. example all OPTIONALs appear below mandatory triple patterns, but I do not see any statements in the text restricting order.

I do not see anything in the grammer (looking mostly at 'GraphPatternNotTriples?') that infers meaning from order.

Lee, am I correct asserting that this is a defect?

Rouben, lowering priority to minor since OPTIONAL ordering can be used to avoid issue.

Wow, I must be really tired. Clearly there is a bug since the handling of the query is inconsistent. That query should do the same thing regardless of the presence of graph2.

I'll fix the rewriters to put all mandatory triple patterns in the 'must' child node of the OPTIONALs node.

Okay, now I'm really confused. When attempting to add this query and it's AST to the TestQueryAndAstStrings I get an WellDesignedOptionalsValidator error on the query if the OPTIONALs are at the top. Why don't I get this error when running the same query from an AnzoClient??

WellDesignedOptionalsValidator was disabled in r254, but is still enabled in MockEngineConfig and others. This should be fixed.

The parser requires OPTIONALs be below mandatory triple patterns to work correctly. Currently, it looks like if this is no done that behavior is largely undefined.

In the parser Group.replaceCurrentContentsWithOptional properly organizes an OPTIONAL with all prior graph patterns in the group. Nothing is done to incorporate later graph patterns into the optional. While this strategy seems to work okay for triple patterns (and maybe others) it doesn't work for the textmatch functional predicate.

SPARQL does specifically group triple patterns before the OPTIONAL as the must part of the OPTIONAL. The must part of the OPTIONAL is (itself) optional. So the parsing behavior is correct. Will look into the meat of the bug now.

(In [2216]) re #316

Remove WellDesignedOptionalsValidator? from all EngineConfigs?.

The proper behavior here is actually to have no results.

OPTIONAL { TP1 } 

is equivalent to:

• TP1 if there is at least one solution to TP1
• {} if there are no solutions to TP1

There is a bug in the Anzo solution generator that handles this incorrectly. I will look at this bug and check in tests for both the correct and buggy cases (now) and a fix for the buggy cases.

(In [2226]) tests and fixes algebra bug surrounding an OPTIONAL with no required part, particularly when the OPTIONAL is the first item in a Group.

(re #316)

(In [2227]) tests and fixes algebra bug surrounding an OPTIONAL with no required part, particularly when the OPTIONAL is the first item in a Group.

(re #316)

Joe, please review the tests & (one line) fix and close.

Others: note the clarification & resolution here: OPTIONALs in SPARQL are order dependent.

reviewer: jpbetz

(In [2228]) remove redundant test in TestQueries? that was replaced by tests in [2227]

(re #316)

Nice! I'm very happy to see this solved.

Thanks!