XUnit Tests Do Not Show Passing or Failing in Test Explorer with DisplayName Specified

[fuzzzerd]

As the title says, tests with the DisplayName specified do not show up correctly in the test explorer; though the actual dotnet test command does appear to run the tests and the console output shows the correct pass/fail numbers.

This works:

[Fact]
public void TestMethod() { Assert.True(true); } 

This does not (though dotnet test output shows tests pass)

[Fact(DisplayName="The Test Method")]
public void TestMethod() { Assert.True(true); } 

[stefanforsberg]

The display name makes xunit behave like nunit/mstest with regards to output from dotnet test -t. This will be fixed with #51.

[stefanforsberg]

Hmm, actually it's a bit more complex then I initially thought so #51 is not in itself enough to make it work unfortunately. If you're willing to work on it we love taking pull requests =)

[Ghostbird]

Might this be related to an issue I have?

[Fact(DisplayName = "Item: Creation")]
public void CreateItem() { }

When test discovery runs I see CreateItem in the list of tests.
When I click the play icon next to CreateItem, the test runs. Once the test passed or failed, I seen an additional test in the list: Item: Creation with the correct status.
The original CreateItem is still in the list and the icon indicates that it was not run.
When I click the play icon next to Item: Creation I get an error message saying that the test was not found.

[mbrankintrintech]

Any update on this? I'm still experiencing this issue and its extremely irritating.

[stefanforsberg]

@mbrankintrintech As stated two posts above your own: "If you're willing to work on it we love taking pull requests".

[stefanforsberg]

@Ghostbird Yes, that is indeed related to this issue. Unfortunately there does not seem to be a clean way to handle this given the current tooling.

[commonsensesoftware]

Expanding as I've discovered the same issue, albeit a different way. This problem also arises if you use the xUnit methodDisplayOptions feature.

@stefanforsberg I agree that the tooling seems pretty weak to properly support this at the moment. I did, however, find that a solution should be possible. I observed that test discovery is using:

dotnet vstest "<assmebly>" /ListFullyQualifiedTests /ListTestsTargetPath:"<temp file>"

This command always lists tests with their fully-qualified names in the form of: <namespace>.<type>.<method>.

I also noticed that this command, which is equivalent to dotnet vsttest /ListTests:"<assembly>",

dotnet test --list-tests

produces the same list, but uses the correct display name of the test.

Possible Short-Term Solution

This issue should be solvable by first outputting file of fully-qualified names and then using something like a zip operation to stitch that together with the display name. Since vstest.exe is responsible for ultimately listing the tests in both cases and lists the tests in the same order, that should work.

Some additional considerations should probably considered to avoid race conditions or file contention. This could be achieved by shadow copying files during discovery. Another option would be to give up if such an error occurs during discovery and restart.

Suggested Long-Term Solution

The TestCase.cs class defined by VSTest definitely supports all of this information (it works in VS after all), but vstest.exe itself does not appear to support output all of it. It might worth filing an issue to the VSTest team for a feature request that would output all of the TestCase information out to a file or - maybe - the console. Perhaps something like a --FullMetadata switch that could be paired with the existing --ListTests and --ListFullyQualifiedTests. For example:

dotnet test --list-tests --full-metadata
dotnet test -t -m
dotnet vstest "<assmebly>" /ListFullyQualifiedTests /ListTestsTargetPath:"<temp file>" /FullMetadata

It might, optionally, be worth asking for a few possible output formats such as *.xml, and *.json (ex: /FullMetadata:json). Since this capability would benefit all VSTest users, the team is likely to consider it. This could also open up the door for future feature enhancements such as supporting xUnit Traits and MSTest Test Categories. Access to TestCase.CodeFilePath and TestCase.LineNumber could also significantly improve Go To Test since symbol resolution would no longer be necessary nor ever ambiguous.

Conclusion

Regardless of approach, some level of refactoring will be required. The current design assumes that test cases are just strings, but they are so much more. At a minimum, the fully-qualified and display names need to be supported.

If I had capacity, I'd love to help drive this with a PR, but - sadly - I don't currently. Hopefully, this research will be useful to someone on this thread to see it through.