From: Xi Wang <xi.wang@gmail.com>
Date: Wed, 21 Dec 2011 10:18:33 +0000 (-0500)
Subject: vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()
X-Git-Tag: v3.2~37^2
X-Git-Url: http://git.openpandora.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=8a78389651b3e411ec5a7df61404734f52d6f4eb;p=pandora-kernel.git

vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()

Commit e133e737 didn't correctly fix the integer overflow issue.

-	unsigned int required_size;
+	u64 required_size;
	...
	required_size = mode_cmd->pitch * mode_cmd->height;
-	if (unlikely(required_size > dev_priv->vram_size)) {
+	if (unlikely(required_size > (u64) dev_priv->vram_size)) {

Note that both pitch and height are u32.  Their product is still u32 and
would overflow before being assigned to required_size.  A correct way is
to convert pitch and height to u64 before the multiplication.

	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls the existing vmw_kms_validate_mode_vram() for
validation.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Reviewed-and-tested-by: Thomas Hellstrom <thellstrom@vmware.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
---

Reading git-diff-tree failed