Wait for data before async recv

[vojtech-frodl]

This fixes an issue where recv() is called on a non-blocking socket immediately after accept() before it's ready for reading, returns -1 and the connection is closed.

I found this when I was trying to profile a Windows executable running in Wine on Linux in a Docker container in Kubernetes on AWS. I tried to connect to it from a local web browser via kubectl port-forward and it kept closing the connection 100% of the times. This was the only setup I was able to reproduce it with. I believe the kubectl port-forward was the main trigger for the issue because I didn't observe it when connecting from inside the Docker container.

[jonasmr]

Hi! Thank you for your submit.

So to understand it correctly, you're saying that you have to call select between the accept and the recv?
To me it looks like the recv is actually supposed to be blocking, but its been a while since I've looked at this.

[vojtech-frodl]

Hi, yes, that's what I observed, select between accept and recv fixed it. If the socket was intended to be blocking, it wasn't in my case, as far as I could tell.

[vojtech-frodl]

Then again, MicroProfileSetNonBlocking(S.ListenerSocket, 1); is being called, so maybe it's supposed to be non-blocking after all?

[jonasmr]

I think this is where it is platform dependent. I'm pretty sure I remember window always creates sockets in Blocking mode.

[vojtech-frodl]

This is my understanding as well:
https://learn.microsoft.com/en-us/windows/win32/winsock/winsock-ioctls#fionbio

When a socket is created, it operates in blocking mode (that is, non-blocking mode is disabled).

The ListenerSocket is created as non-blocking here:

S.ListenerSocket = socket(PF_INET, SOCK_STREAM, 6);
MP_ASSERT(!MP_INVALID_SOCKET(S.ListenerSocket));
MicroProfileSetNonBlocking(S.ListenerSocket, 1);

Then MpSocket Connection = accept(S.ListenerSocket, 0, 0); creates a new socket Connection which is also non-blocking:
https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-accept#remarks

The newly created socket is the socket that will handle the actual connection; it has the same properties as socket s

[jonasmr]

Hi.

Can you try this code, and see if this fixes your issue?

The intent of the code here is that the socket should be blocking, but it seems like it isn't.
These changes should fix it up, and hopefully let you continue

diff --git a/microprofile.cpp b/microprofile.cpp
index b5a86cc..f37ea13 100644
--- a/microprofile.cpp
+++ b/microprofile.cpp
@@ -8411,6 +8411,7 @@ bool MicroProfileWebServerUpdate()
 	if(!MP_INVALID_SOCKET(Connection))
 	{
 		std::lock_guard<std::recursive_mutex> Lock(MicroProfileMutex());
+		MicroProfileSetNonBlocking(Connection, 0);
 		char Req[8192];
 		int nReceived = recv(Connection, Req, sizeof(Req) - 1, 0);
 		if(nReceived > 0)
@@ -8470,7 +8471,7 @@ bool MicroProfileWebServerUpdate()
 				{
 				case EMICROPROFILE_GET_COMMAND_LIVE:
 				{
-					MicroProfileSetNonBlocking(Connection, 0);
+					//MicroProfileSetNonBlocking(Connection, 0);
 					uint64_t nTickStart = MP_TICK();
 					send(Connection, MICROPROFILE_HTML_HEADER, sizeof(MICROPROFILE_HTML_HEADER) - 1, 0);
 					uint64_t nDataStart = S.nWebServerDataSent;
@@ -8509,7 +8510,7 @@ bool MicroProfileWebServerUpdate()
 				case EMICROPROFILE_GET_COMMAND_DUMP:
 				{
 					{
-						MicroProfileSetNonBlocking(Connection, 0);
+						//MicroProfileSetNonBlocking(Connection, 0);
 						uint64_t nTickStart = MP_TICK();
 						send(Connection, MICROPROFILE_HTML_HEADER, sizeof(MICROPROFILE_HTML_HEADER) - 1, 0);
 						uint64_t nDataStart = S.nWebServerDataSent;

[vojtech-frodl]

Confirming that your fix also works.

[vojtech-frodl]

Hello again Jonas,

I re-tested your fix on Windows (no Wine or Docker), and I found that, with a small probability, it causes my program to hang for about 80 seconds. The callstack during the hang is:

ntdll.dll!NtWaitForSingleObject()
mswsock.dll!SockWaitForSingleObject()
mswsock.dll!WSPRecv()
ws2_32.dll!recv()
1-MicroProfile.dll!MicroProfileWebServerUpdate()
1-MicroProfile.dll!MicroProfileFlip_CB(void *)
1-MicroProfile.dll!MicroProfileFlip(void *)
1-MicroProfile.dll!Profiler::EndFrame_v0()
DiagManager.dll!Diag::DiagManager::EndFrame_v0()
[...]

I recorded a video of the MicroProfile page during this wait:
https://www.youtube.com/watch?v=Hwndhx8ertg

Interestingly, if I refresh the MicroProfile page during the hang, it connects immediately, which seems like a temporary workaround.

The repro rate is fairly low, but I noticed it’s more likely to get stuck if my program isn’t collecting much profiling data at the time I try to open the connection.

Also, I see the same blocking behavior with both your fix and my original attempt, the only difference is that in my version it blocks in select() instead of recv().