Image Depth Maps Tutorial for iOS: Getting Started

toojuice · January 25, 2018, 6:12am

@doonoo let me look into this more closely. I’ll report back soon.

toojuice · January 31, 2018, 8:50am

So while playing around, I noticed the same thing you did. I believe the HEIC filetype has the ability to store multiple images in the container. So it could be that the main image is the one with the blurred background.

However, the depth data is always available, if the image has the blurred background or not.

I did this experiment to prove it to myself:

Take a portrait mode photo.
Use PHImageManager with PHAsset to read the last photo taken.
Display the photo and the depth data in a UIImageView
Go into the Photos app
Edit the last photo to remove the portrait effect
Open my test app and see both the unblurred photo and the depth data

I think maybe a next step for you would be to look into the HEIC image format and see if you can read the original and the portrait photo from the same container. I’m guessing it might be there.

Another interesting test would be to change your camera to take pictures as JPGs. Then it might actually save two separate images – one for the original and one for the portrait.

You can access this by going Settings App → Camera → Formats → Most Compatible

zhanqan · February 22, 2018, 11:21am

Hi,thanks for contributing!
Now,I have a question,Can we get the point cloud data from the picture and how? Hope for your answering!

shogunkaramazov · February 22, 2018, 4:28pm

@toojuice Do you have any feedback about this? Thank you - much appreciated! :]

toojuice · February 22, 2018, 7:48pm

Hi @zhanqan,

There is no point cloud for depth data as it isn’t obtained using a single depth sensor. It’s using two slightly offset cameras to imitate stereoscopic vision. Just to be clear, this tutorial is not about the camera and sensor used for FaceID. This is about the depth data you can get from dual cameras on the back of an iPhone 7 Plus, iPhone 8 Plus, and iPhone X.

eyzuky · March 7, 2018, 1:09pm

Not relevant anymore, thanks for your awesome reply

wedouglas · March 7, 2018, 3:32pm

Hi,

I was wondering if you have an information on what values the raw depth data takes on. I know that when converting to greyscale, the depth data at the individual pixel level has a value between 1 and 255. However, from my understanding the raw normalized depth data is actually anywhere from 0 to >1 for object closer than 1 meter. If the raw data does in fact have much more range than 1/255, I feel like it could be manipulated for better results.

Any insight on the topic would be a great help! Thanks!

shogunkaramazov · March 8, 2018, 3:20am

@toojuice Can you please help with this when you get a chance? Thank you - much appreciated! :]

toojuice · March 8, 2018, 8:07am

Hi @wedouglas,

Yes, you’re right. The raw disparity data can be greater than 1. That’s the reason we normalize the data before using it to create a mask.

Take a look at the code for normalize() in CVPixelBufferExtension.swift. You can see here that first the minimum and maximum disparity values are found and then used to normalize all pixels to fall in the range between 0.0 and 1.0.

Other options you could take would be to clamp and values to a maximum of 1.0, but you will lose a little bit of depth information that way. There are perfectly good reasons to do this, though.

wedouglas · March 8, 2018, 6:06pm

Yeah, that makes sense. I was thinking that maybe the depth values could be better manipulated if you were trying to focus on a very specific slice of distance in the image. For example, the raw depth data isn’t terribly linear, so 0 to .5 in raw data corresponds to a much greater physical distance than .5 to 5. Therefor maybe it could better be manipulated such that your mask uses 1-255 for a very narrow slice of physical space, and just 0 for everything else, rather than spreading 0-255 over the entire image.

One more question. Do you know if there is an easy way to save the greyscale depth representation as an image itself? I tried saving the UIImage and CIImage to the library, but it doesn’t seem to work. Maybe that’s it’s because those actually don’t contain the real pixel data to create the image from?

Any suggestions?

toojuice · March 8, 2018, 10:21pm

How did you try to save the image? Using UIImagePNGRepresentation?

seereen2004 · March 9, 2018, 4:37pm

How to get the depth image using Truedepth camera in IphoneX ? … As I know it is not stereo camera … can you have tutorial to extract the depth images using this camera and store it ?

wedouglas · March 11, 2018, 9:02pm

I’ve tried using both PNG and JPEG representation like that. It seems like those aren’t actually getting any data or that the data is unsupported. From the documentation for UIImagePNGRepresentation:

A data object containing the PNG data, or nil if there was a problem generating the data. This function may return nil if the image has no data or if the underlying CGImageRef contains data in an unsupported bitmap format.

I believe I’m out of my depth here, no pun intended. Is there something about the data in the cvpixelbuffer that isn’t supported by a jpeg/png? If that’s the case, any suggestion on how to convert this data accordingly?

toojuice · March 12, 2018, 3:15pm

Ok. Let me experiment a little and I’ll get back to you. I haven’t tried this out, yet.

toojuice · March 12, 2018, 7:40pm

I’ve done a little digging and the reason UIImagePNGRepresentation is returning nil is because the UIImage for the depth data map is not based on a CGImage. It was created from a CIImage, which was created from a CVPixelBuffer.

If you want to be able to save the depth data map, you’ll need to first create a CGImage and then create the UIImage from that. When you create a CGImage, you’ll have to be careful with the orientation of the data, so keep that in mind.

seereen2004 · March 15, 2018, 6:46pm

@toojuice Can you post a tutorial about getting depth map using Truedepth camera in iphoneX?
I just want to read the image and save the depth… can you help me on this please
Thank you

toojuice · March 16, 2018, 7:19pm

Thanks for the suggestion! I’ve forwarded the idea along to Ray :]

seereen2004 · March 20, 2018, 7:55pm

@toojuice I tried this code and I added other pictures that I captured using Portrait mode in iphone X with name test10.jpg this program only read test01.jpg to test09.jpg not more !
In addition, I can only see the half top part of the image in original mode ! … do you know why?
Also I would like to save the depth image before normalize the colors … how can I do it? … appreciate your help

eyzuky · March 21, 2018, 1:17pm

Hi, in my app we are implementing a Portrait mode for blurring only the background of an image.
I attach my result.

In my result the edges of the lamp are not sharp… I tried playing with all the parameters of the mask input (width, slope) and I cannot get an accurate result.
I assume maybe Apple are doing some more post process on the Depth Data, but I am not sure. Maybe I am missing something obvious?
Would appreciate your input.

Thanks!

toojuice · March 22, 2018, 8:44pm

Adding a test10.jpg to the project works for me. Make sure that the extension is lowercase… i.e. .jpg and NOT .JPG

As for saving the depth image, you’ll need to first create a CGImage and then create the UIImage from that. When you create a CGImage, you’ll have to be careful with the orientation of the data, so keep that in mind.