Problem related to Split screen metal rendering

[argha92]

I am trying to render a split screen (left + right) . On the left I am rendering a pyramid 3D and on the right I am trying to render the depth texture corresponding to the pyramid. I have created a depthTexture and did a blind pass then I am trying to render the texture on a quad. However the texture is not getting rendererd.

I tried to see if the quad is coming properly by rendering it in sollid color and it works. I also saw the uv corresponding to the quad is coming correctly.

 func draw(in view: MTKView) {
        guard let commandBuffer = Self.commandQueue.makeCommandBuffer() else { return }
        
        
        guard let drawable = view.currentDrawable else { return }
        
        // Get the render pass descriptor
        guard let descriptor = view.currentRenderPassDescriptor else { return }
        
        guard let depthCommandBuffer = Self.commandQueue.makeCommandBuffer(),
              let depthRenderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: depthRenderPassDescriptor) else { return }
                
        let viewportForDepth = MTLViewport(
            originX: 0,
            originY: 0,
            width: 2048,
            height: 2048,
            znear: 0,
            zfar: 1
        )
        depthRenderEncoder.pushDebugGroup("Depth Pass")
        depthRenderEncoder.setViewport(viewportForDepth)
        // Configure matrices for right cube (top view)
        let viewMatRight = getViewMatrix(
            eye: SIMD3<Float>(0, 3, 0),
            center: SIMD3<Float>(0, 0, 0),
            up: SIMD3<Float>(0, 0, -1)
        )
        
        let projMatRight = createProjectionMatrix(
            fov: 60.0 * .pi / 180,
            near: 0.0001,
            far: 3.0,
            aspect: 1.0
        )
                
        uniforms.vMatrix = viewMatRight
        uniforms.pMatrix = projMatRight
        
        depthRenderEncoder.setDepthStencilState(depthStencilState)
        depthRenderEncoder.setRenderPipelineState(depthPipelineState)
        depthRenderEncoder.setVertexBuffer(positionBuffer, offset: 0, index: 0)
        depthRenderEncoder.setVertexBuffer(normalBuffer, offset: 0, index: 1)
        depthRenderEncoder.setVertexBytes(&uniforms, length: MemoryLayout<Uniforms>.stride, index: 2)
        depthRenderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 36)
        depthRenderEncoder.popDebugGroup()
        // Finish encoding
        depthRenderEncoder.endEncoding()
        // END OF TEXTURE GENERATION

        // Create a command buffer
//        guard let commandBuffer = Self.commandQueue.makeCommandBuffer() else { return }

        // Set up common render pass properties
        descriptor.colorAttachments[0].clearColor = MTLClearColor(red: 1, green: 1, blue: 1, alpha: 1) // White
        descriptor.colorAttachments[0].loadAction = .clear  // Clears once at the start
        descriptor.colorAttachments[0].storeAction = .store // Store the result

        // ---- Render Pyramid to Left Screen ----
        if let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: descriptor) {
            let rotationMatrix = matrix_rotation_y(angle)
            
            renderEncoder.setDepthStencilState(depthStencilState)
            renderEncoder.setRenderPipelineState(pipelineState)
            renderEncoder.setVertexBuffer(positionBuffer, offset: 0, index: 0)
            renderEncoder.setVertexBuffer(normalBuffer, offset: 0, index: 1)

            // Left Viewport for Pyramid
            let viewportLeft = MTLViewport(
                originX: 0,
                originY: 0,
                width: Double(view.drawableSize.width / 2),
                height: Double(view.drawableSize.height),
                znear: 0,
                zfar: 1
            )
            renderEncoder.setViewport(viewportLeft)

            let viewMatLeft = getViewMatrix(
                eye: SIMD3<Float>(0, 3, -5),
                center: SIMD3<Float>(0, 0, 0),
                up: SIMD3<Float>(0, 1, 0)
            )
            let aspectRatioLeft = Float(view.drawableSize.width / 2) / Float(view.drawableSize.height)
            let projMatLeft = createProjectionMatrix(
                fov: 60.0 * .pi / 180,
                near: 0.0001,
                far: 10.0,
                aspect: aspectRatioLeft
            )

            uniforms.mMatrix = rotationMatrix
            uniforms.mITMatrix = computeNormalMatrix(from: rotationMatrix)
            uniforms.vMatrix = viewMatLeft
            uniforms.pMatrix = projMatLeft

            renderEncoder.setVertexBytes(&uniforms, length: MemoryLayout<Uniforms>.stride, index: 2)
            renderEncoder.setFragmentBytes(&uniforms, length: MemoryLayout<Uniforms>.stride, index: 0)
            renderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 36)

            // End first render pass
            renderEncoder.endEncoding()
        }


        // ---- Render Quad to Right Screen ----
        descriptor.colorAttachments[0].loadAction = .load  // Clears once at the start
        descriptor.colorAttachments[0].storeAction = .store
//        depthRenderPassDescriptor.depthAttachment.loadAction = .load
//        depthRenderPassDescriptor.depthAttachment.storeAction = .store

        if let quadRenderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: descriptor) {
            let viewportRight = MTLViewport(
                originX: Double(view.drawableSize.width / 2),
                originY: 0,
                width: Double(view.drawableSize.width / 2),
                height: Double(view.drawableSize.height),
                znear: 0,
                zfar: 1
            )
//            quadRenderEncoder.setTriangleFillMode(.lines)
            quadRenderEncoder.setViewport(viewportRight)
            quadRenderEncoder.setRenderPipelineState(renderTexturePipelineState)
            quadRenderEncoder.setVertexBuffer(quadVertexBuffer, offset: 0, index: 0)
            quadRenderEncoder.setVertexBuffer(quadUvBuffer, offset: 0, index: 1)
            quadRenderEncoder.setFragmentTexture(depthTexture, index: 2)
            quadRenderEncoder.drawIndexedPrimitives(type: .triangle, indexCount: 6, indexType: .uint16, indexBuffer: quadIndexBuffer, indexBufferOffset: 0)

            // End second render pass
            quadRenderEncoder.endEncoding()
        }

        // Present and commit command buffer
        commandBuffer.present(drawable)
        commandBuffer.commit()
        commandBuffer.waitUntilCompleted()
        
        
    }

This is my draw call. And this is my fragment shader corresponding to the quad:

fragment float4 tex_fragment_main(QuadOut in [[stage_in]],
                                  depth2d<float, access::sample> depthTexture [[texture(2)]]) {
//    constexpr sampler textureSampler(coord::normalized, address::clamp_to_edge);
    constexpr sampler smpler(
      coord::normalized, filter::linear);
    float2 texCoord = in.uv ;
    float d = depthTexture.sample(smpler, texCoord);
    return float4(d, d, d, 1.0);
//    return float4(in.uv.x, in.uv.y, 0.0, 1.0);
}

Kindly let me know if you can find the issue.

[caroline]

There are many things that can go wrong, and I can’t see anything wrong by looking at your code - I’d have to run it.

Do you have a minimal project? You probably can’t add a zip file here, but maybe you could upload a project to GitHub or Dropbox?

[argha92]

Stenciling 2.zip

Hi, Here is the project @caroline

[caroline]

The depth texture might look black (0) and white to you (1), but in reality, the black value is 0.9999.

In text_fragment_main, add this before the return.

if (d < 1.0) {
  d = 0;
}

And then you’ll see your depth texture.

The depth value of 0.9999 has to do with the near and far values.

If you remove the above if statement, and change the depth’s near and far to:

let projMatRight = createProjectionMatrix(
    fov: 60.0 * .pi / 180,
    near: 0.1,
    far: 100.0,
    aspect: 1.0
)

Then you’ll get this result:

Where the depth values show up on the render, because they are relatively further back in the scene.

The depth buffer is non-linear. There’s a small bit of code in Chapter 22: Reflection & Refraction, Section 7. Adding Smoothness using a Depth Texture, where the non-linear depth is converted to a linear value.

I hate to say it: I don’t fully understand the math nowadays - it’s been a while since I looked into linearizing depth buffers, but this link might explain it: Linear depth buffer for rendering shadows - Graphics and GPU Programming - GameDev.net

This article from long ago suggests that you should put your near value as far away from the camera as possible: Learning to Love your Z-buffer.. Yours at 0.0001 is very close.

[argha92]

But even then it is incorrect. I expect to see a gradient B/W image. Like the one you are seeing in the metal profiler. I am getting a B/W boolean texture

.

I am expecting the depth texture to lie between 0-1. Or are there some other range? @caroline

[caroline]

I edited my previous message to add a little bit about the non-linearity of the depth buffer, which is indeed from 0 - 1. And I suggested you change the near / far values if possible.

[caroline]

I think the gradient you are seeing is not measurable by the GPU. It’s normalised the values so that you can see them, but if you run the cursor over the texture values, they are all 0.9999.

[argha92]

You are right. Metal is trying to normalize and show the results.