Fast source-based test discovery with OmniSharp

[felix-b]

Guys, kudos for taking on the challenge! If you're aimed at creating an excellent testing experience in VS Code, I'd like to join.

After passing through the code, here is my two cents. Running dotnet test for test discovery will be slow for seriously-sized projects. The UX will be identical to that of Visual Studio test runner, which is not that wonderful compared to ReSharper, IMHO.

According to a research I did earlier, OmniSharp server is capable of scanning source file contents, and providing a list of features on each method. Specifically, we're going to look for XunitTestMethod, and maybe NUnitTestMethod in the future. (See this PR).

We can spin up background discovery for entire project/solution, as soon as we detect that .NET Core project/solution is loaded. We can re-scan the active file as it is being edited.

Discovery based on source code can also bring me the tests even if my code doesn't compile - at least it will allow exploring and navigating the code. In Visual Studio test runner, the entire solution must be compiled before test discovery can start, so that it sometimes takes more than a minute to see my tests - this one is definitely a UX killer.

[formulahendry]

Hi @felix-b , thanks for you passion and welcome to join! Agree with you that dotnet test is a little heavy for large project. PR, discussion or suggestion are always welcome! Let's make the testing experience better and better. 😄

[ciel]

I would love to do this, but I can't find any documentation online of how to integrate OmniSharp with an extension.

[ciel]

The best I can find is the npm package omnisharp-client but I can't find much on how to use it.

[felix-b]

My initial thought was going the same way as omnisharp-vscode does - look at server.ts from their repo.

But then I came across the omnisharp-client package, too, and it seems like the latter is the way to go. API based on the concept of observables is exactly what we need here. I didn't yet have a chance to try to figure out how to use it, but you can try these:

• readme.md in the omnisharp-node-client repo
• omni.ts in the omnisharp-atom repo, which appears to be the only official user of omnisharp-node-client. Both repos belong to OmniSharp - probably that's why there's no documentation, they built it for their own needs.

Technically, as far as I understand, the only supported way to interact with the roslyn-server is spawning it as a child process, and interacting with it through stdin/stdout.

I hope to spend a few hours on it over the next week.

[janaka]

@felix-b @ciel you might be interested in PR #83.

[dgioulakis]

Curious if there's any progress with this. Just went through the source for test discovery and found it's dependent upon executing and parsing both dotnet test & dotnet vstest.

[janaka]

@Cephei thanks for chasing up on this. The PR #83 which I tagged in my last comment ^ is the latest and represents latest thinking. The idea is that VS Test Platform is what Visual Studio's test explorer is build upon hence makes sense to build upon that in VS Code also. Further the dotnet test and dotnet vstest commands are wrappers around the VS Test Platform but obviously executing the CLI each time isn't efficient.

That PR is a spike before we dive into a proper implementation. TBH I've not been able to spend any time on it for a while but hoping to get my head back into it soon.

[stefanforsberg]

Do we have any hard data on how much faster a discovery of say 10,100 and a 1000 tests would be with the approach in #83?

[janaka]

Not got to that yet. Curious why you picked 10,100 as a number :). Definitely something to test out though.

Even if it's the perf is the same as the cli approach, at present I feel depending on a versioned API/protocol is a more solid than a cli + reporting format that aren't strictly versioned from what I understand. Might be wrong on that though.