Setting a custom capture framerate

[Boscop]

I bought a webcam that can do 30ps Full Hd. I tested it in OBS-Studio, at first I got very low capture framerate, about 5 fps. But then I read this post and it worked: By setting the "Resolution/FPS Type" to Custom and setting Resolution to 1920x1080 I got the 30 FPS capture framerate.

Then I tried to use the webcam in my opengl application, capturing it with Escapi and rendering it to the screen.
But in my application I ALSO only get that low capture framerate with Escapi :(
(Btw, I get high FPS with a friend's webcam so it's not my application's fault.)
I talked on IRC with the developer of OBS about what OBS does when you set a custom capture FPS, it uses DirectShow functions to change the pin type. He pointed me to this:
https://github.com/jp9000/libdshowcapture/blob/master/source/device.cpp#L304

Could the same be done with Windows Media Foundations?
It seems that some cameras (like mine) use a lower capture framerate by default than what they could, so it would be useful to be able to set a custom capture framerate with Escapi, too :)

[jarikomppa]

Sorry, no idea. If someone does figure this out, pull requests are welcome.

…

On Wed, Sep 6, 2017 at 3:17 PM, Boscop ***@***.***> wrote: I bought a webcam that can do 30ps Full Hd. I tested it in OBS-Studio, at first I got very low capture framerate, about 5 fps. But then i read this post <https://obsproject.com/forum/threads/getting-the-most-out-of-your-webcam.1036/> and it worked: By setting the "Resolution/FPS Type" to Custom and setting Resolution to 1920x1080 <https://i.imgur.com/9JECr6q.png> I got the 30 FPS capture framerate. Then I tried to use the webcam in my opengl application, capturing it with Escapi and rendering it to the screen. But in my application I ALSO only get that low capture framerate with Escapi :( I talked on IRC with the developer of OBS about what OBS does when you set a custom capture FPS, it uses DirectShow functions to change the pin type. He pointed me to this: https://github.com/jp9000/libdshowcapture/blob/master/ source/device.cpp#L304 Could the same be done with Windows Media Foundations? It seems that some cameras (like mine) use a lower capture framerate by default than what they could, so it would be useful to be able to set a custom capture framerate with Escapi, too :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEQ_Rw2sv_TWhafdXmMfKDKDg8PYKVETks5sfo1SgaJpZM4POSC7> .

[codec-abc]

I can help a bit since I am working on the same subject. The first thing you need to watch for is to find the supported "capture mode". What I call capture mode is a combination of encoding, (like RGBA32, MJPG, NV12, etc..) width, height and framerate. For that, I recommend to use a tool like graph studio next. Then, you have to note a few things:

1. Escapi only support a few encoding. For example, MJPG is not supported and on my laptop this is only encoding running at 720p at 30 fps.
2. Escapi does not care about framerate. If it finds 2 "capture mode" that only differs and encoding and framerate it will pick the last it encounter (if I recall correctly). I have tweaked the library a bit so it manage it well enough for my purpose. I can guide you through some code if you need to.
3. There is post processing to convert the capture buffer (regardless of its type) to RGBA32 format. This is done only by the CPU without SIMD instructions and I think it can put some heavy workload on the CPU at high resolution and framerate.

[Boscop]

I'd really appreciate your help!

1. I downloaded and ran graphstudionext, how can I figure out the possible capture modes?
   I clicked on "add filter" and it showed 2 entries for my camera, which seems to have the same data.
   Both look like this:
   
   Why do both have MERIT_DO_NOT_USE?

2. I'm interested to know how you tweaked the escapi lib to allow you to choose a different capture mode. It's this code, right?
   codec-abc@879ed81#diff-c2dd74b222de2e25dcac565744140e6f

3. Yes, I'd appreciate if it was possible to treat the given resolution as a maximum resolution, so that it chooses the highest resolution width * height <= max_width * max_height (still fitting in the buffer / having an upper bound for the buffer size) and then it would return the found width and height, so I could do the scaling on the GPU automatically. (Creating a texture with the returned width and height, and uploading that when drawing the quad.)
   Do you have any plans to optionally eliminate the CPU heavy rescaling code in your Escapi fork?

Btw, I've been using my own Rust bindings, which I started before Escapi had official Rust bindings.
They are closer to the C API: https://github.com/Boscop/escapi-rs
The reason I'm still using them is that it's possible to get a camera's name without starting capturing with it, which I need for choosing the right camera by name before I start capturing...

[codec-abc]

1. You are almost there: Add the filter to the graph then right click the "Capture" pin and click "Properties" and then you will probably have a popup where you can choose the encoding, the resolution and the framerate. You need to try every combination to know what your camera can offer on your computer (the same camera might have different option with 2 different computer/OS). Each combination represent a Media SubType (This is the reason there is so many). You can do in the code too, by enumerating all the Media SubType and finding its properties (but you will need to find your way through the Windows API which is not really easy). About the MERIT_DO_NOT_USE I have no idea what it is.
2. Yes that is part of it, but I think I will change it to give directly an ID to map to a media sub-type.
3. I have been partly wrong on my previous remark. There is a raw capture mode in escapi (not sure if it is used though) that would allow you to bypass the conversion. Still you will need to implement the conversion it in a shader if you want to display it properly, and depending on the encoding chosen it might be difficult (impossible maybe). For instance, converting NV21 to RGB is easy in a shader but doing the same with MJPEG might not even be possible. By the way, you might want to change the code that choose the capture mode like I did here so it suits your needs. For me, I chose to convert the buffer back to a YUV format (because I need a luminance buffer to do some image processing stuff, and it is still easy to display). I need to benchmark it but if it run well enough I might leave the default conversion (I only target 720p resolution at 30 fps). And the code does not do rescaling only conversion. The author chose to output the image as an RGBA32format no matter what the Windows API throw at him (except in raw capture mode). From my point of view, it is sensible default (except maybe that RGB24 would have been better than RGBA32) but in some case the overhead of the conversion may be too high for certain use cases.

[Boscop]

I was quite busy lately.. Now I tried that and when I open the properties, I get this:

It doesn't let me choose a different framerate than 5 for YUY2.
But it also has MJPG output. When I choose that, it sets the frame rate field to 30:

Btw, here it lets me change the framerate with the up/down buttons, I also can get 25, 20 and 5 fps.

So that means YUY2 is the first / default capture mode, and Escapi chooses the default / first?
And now, how can I get MJPG @ 30fps with your escapi fork?

Thanks so much!

Btw, any idea why my camera appears twice in Filters?

Btw, IMO Escapi should convert from the camera format to a consistent output format that can be easily scaled in a shader but it shouldn't scale it on the CPU.
So I think that RGB24 (and the highest resolution that fits into the given buffer) is the most reasonable default. So it would also return the width and height that it found as the highest combo that fits into the buffer (width * height * 3 <= given_buffer_size).

If you want, we can work on a fork together that does this. It could also have the raw mode that you want, but I think for most usecases the unscaled RGB24 mode is the most appropriate.

[codec-abc]

It seems that your computer and/or camera is not able to stream in YUY2 @ 1920x1080 at 30 FPS. This is not surprising to be honest. Note that if you lowered the resolution you might have been able to chose a higher framerate in YUY2. I think this is a bandwidth problem. YUY2 is probably less space efficient that MJPG so it consumes more bandwidth and you have probably reached the limit at 5fps (for 1980x1080).

If your goal is to display and/or process the camera stream at 1980x1080 you will need to "tweak" Escapi. My fork is kind of ugly (and I tweaked some more locally, I might push those changes later) and not there yet, but I can give you some pointers.

The first thing to do is to implement the conversion for the MJPG encoding -You might want to double check if there is not another format that may be able to stream your camera at 1920x1080@30fps because MJPG is not easy to support- To do that you should add the MJPG format to the list of supported format in escapi (like here). But then you have to provide a conversion function. The conversion function should take the MJPG buffer of each frame and convert it to RGBA. Unless you planned to write a JPEG decoder yourself (which is a quite hard task), you might want to use a library. This one should work and if you want to understand the reason of the fork, look at this issue. Maybe libjpeg would work too, but if you want to find out you will have to test it by yourself. :)

At this point, you just need to make sure that Escapi selects the correct capture mode like I explained before (point 3 of my previous message). For your case you should select the capture mode with MJPG encoding, with a resolution of 1920x1080 and 30 frames per second. For testing purpose you might want to hardcode the capture mode selection. This is currently the part I want to improve on Escapi. I want to return each possible capture mode and be able to select it using some kind of id. It is a bit complicated since I want to do it from another language (C# in my case).

My camera shows twice as well in GraphStudioNext and I have no idea why, but it does not seems important.

EDIT: If you manage to do that part cleanly, you might want to summit a pull request because I will welcome the addition.

[Boscop]

I tried with your fork at this commit but when I give it a desired framerate of 30, or 20 I only get a completely blue cam buffer from Escapi. Only when I give 5 for the desired framerate it works like before.
Is it trying to use YUY2 with the given framerate (30)?

[codec-abc]

I don't think you should use my fork (at least in this state). Especially since it does not support MJPG. I don't know what happen for sure but I think Escapi does not find compatible capture mode at 1980x1080@30 fps.
EDIT:
Well, I suppose you can use my fork to test stuff if you are not afraid :). If you add fake MJPG support (by doing a dumb buffer copy) like here you should have some more "meaningful" output. It should be RGBA noise. Then, if you dump this noise in a file and save it with a .jpg extension GIMP should be able to display the image.

[codec-abc]

About your remarks on the output mode I am not sure I agree. Ideally I would like things to work that way:

1. I ask Escapi to send me a list of connected camera with their available capture mode. For each capture mode, Escapi tells me if it would be able to convert it to a default format (let's say RGB24 for instance).
2. I chose a camera and a capture mode (or maybe even let Escapi pick one automatically) and tell Escapi if I want the default conversion to happen.
3. When a frame is completed Escapi send me either the raw buffer or a converted buffer depending on what I asked at the previous step. Then, what I do with that buffer is up to me. For my use case I will probably chose the raw buffer because I don't want additional CPU cost.

This way, the API is easy to use and the entry barrier quite low (for the case where the default suit you). But in case you want control you still have it without tweaking the library internals.

About the fork, if you want something that only works in your case if might not fit this library's goals. But I don't think there is a need to do it (maybe except temporary, for testing stuff). @jarikomppa seems to welcome pull requests, and I think if you come with something that works well and is simple to use he will accept the PRs. Anyway, if you need help I will try my best (in the little time I have).

[Boscop]

Ah yes, getting a list of supported capture modes from escapi would be fancy.

So it would work if I just get the MJPG buffer out and then use the jpeg-decoder crate to decode every frame at the client side?

Is this what you are planning to do with your fork already?

Btw, in your TransformImage_MJPG you can do the same as in TransformImage_RGB32, i.e. MFCopyImage(aDest, aDestStride, aSrc, aSrcStride, aWidthInPixels * 4, aHeightInPixels); which will probably be faster (using memcpy). And you increased gConversionFormats from 4 to 6 but only added 1 more format (MJPG) to the gFormatConversions array. So gConversionFormats should probably be 5.

If I return the raw MJPG buffer, can it be larger than the decoded frame would be in RGB32 (size of user-provided buffer)?
If not, what was the reason why you added the bufferLength arg to all conversion functions?

[codec-abc]

Yes I think it should work if you had the MJPG as a supported format and then send the buffer through the jpeg-decoder crate. This is indeed what I did with my fork (among other things, like getting the available capture mode). You just need to be cautious as the buffer length will probably vary at each frame because of the compression. I think it will always be smaller than the allocated one by Escapi (which should be 1920x1080x4 bytes) so you are pretty safe about access violation stuff. Yet, I am not sure that if you pass trailing (garbage) bytes to the jpeg-decoder it will work properly. So just give it the exact numbers of bytes took by the JPG frame.

The code of my "fork" is indeed neither clean nor optimal that is why I tried to explain you the overall pictures before sending you piece of code :) . memcpy make sense for copying the MJPG buffer (and I think I use it in my more up to date local version) but MFCopyImage is at best confusing. It gives the illusion of dealing with pixels while you are dealing with compressed data.

[Boscop]

Ah I see. Yea, it would be cool to get MJPG support working. Not sure if the MJPG frame has header info that tell the decoder how long the compressed frame data is. If it has a header, we can just always copy the whole buffer.

[codec-abc]

You might want to look at that line. It seems that I found a way to find out the actual MJPG buffer length. Sadly, I don't remember anymore and my code being poorly documented it is not very helpful.
EDIT: Indeed, calling Lock on the IMFMediaBuffer retrieve the length of the valid data in the buffer. (source)

[Boscop]

I just tried with this function (same as TransformImage_RGB32):

void TransformImage_MJPG(
	BYTE*       aDest,
	LONG        aDestStride,
	const BYTE* aSrc,
	LONG        aSrcStride,
	DWORD       aWidthInPixels,
	DWORD       aHeightInPixels
	)
{
	MFCopyImage(aDest, aDestStride, aSrc, aSrcStride, aWidthInPixels * 4, aHeightInPixels);
}

And I get a access violation when it calls mConvertFn in CaptureClass::OnReadSample().
Hm, why would it cause access violation?

[codec-abc]

No idea but like I said MFCopyImage does not make sense for MJPG buffers. You should try with

memcpy(aWidthInPixels * aHeightInPixels * 4);

instead. But you will have "garbage" bytes after the end of the actual JPG buffer. I recommend that you look at my previous message and get the correct length of the buffer.