Subject: Broken regex in nvi on several platforms

From: Alexander Viro <[email-masked]>
Date: Wed, 12 Feb 2003 06:09:45 -0500 (EST)

Package: nvi
Version: 1.79-20
Severity: important
Tags: patch

	On any platform with sizeof(unsigned long) > sizeof(int) = 4
\<, \> and \| in regular expressions do not work.  Reason: definitions
of OBOW and friends in regex/regex2.h are b0rken.

What happens is unexpected sign expansion in equivalents of
	(unsigned long)(19 << ((unsigned)27))
Spencer had mistakenly assumed that casting the second argument of << to
unsigned would make the result unsigned.  It doesn't work - signedness of
bitwise shift result is determined by signedness of the first argument
alone.

The bug is easy to demonstrate - on e.g. alpha start vi on a non-empty file,
say
:se ext
/(A|B)
and see
RE error: "can't happen" -- you found a bug
The same bug breaks ^A on the same platforms - you always get "pattern not
found".

Trivial fix follows (lifted from *BSD - they have Spencer-derived regex in
libc, so they had to deal with the same bug):

diff -urN nvi-1.79/regex/regex2.h nvi-1.79-new/regex/regex2.h
--- nvi-1.79/regex/regex2.h	1996-07-09 19:05:31.000000000 -0400
+++ nvi-1.79-new/regex/regex2.h	2003-02-12 05:10:09.000000000 -0500
@@ -85,28 +85,28 @@
 #define	SOP(op, opnd)	((op)|(opnd))
 /* operators			   meaning	operand			*/
 /*						(back, fwd are offsets)	*/
-#define	OEND	(1<<OPSHIFT)	/* endmarker	-			*/
-#define	OCHAR	(2<<OPSHIFT)	/* character	unsigned char		*/
-#define	OBOL	(3<<OPSHIFT)	/* left anchor	-			*/
-#define	OEOL	(4<<OPSHIFT)	/* right anchor	-			*/
-#define	OANY	(5<<OPSHIFT)	/* .		-			*/
-#define	OANYOF	(6<<OPSHIFT)	/* [...]	set number		*/
-#define	OBACK_	(7<<OPSHIFT)	/* begin \d	paren number		*/
-#define	O_BACK	(8<<OPSHIFT)	/* end \d	paren number		*/
-#define	OPLUS_	(9<<OPSHIFT)	/* + prefix	fwd to suffix		*/
-#define	O_PLUS	(10<<OPSHIFT)	/* + suffix	back to prefix		*/
-#define	OQUEST_	(11<<OPSHIFT)	/* ? prefix	fwd to suffix		*/
-#define	O_QUEST	(12<<OPSHIFT)	/* ? suffix	back to prefix		*/
-#define	OLPAREN	(13<<OPSHIFT)	/* (		fwd to )		*/
-#define	ORPAREN	(14<<OPSHIFT)	/* )		back to (		*/
-#define	OCH_	(15<<OPSHIFT)	/* begin choice	fwd to OOR2		*/
-#define	OOR1	(16<<OPSHIFT)	/* | pt. 1	back to OOR1 or OCH_	*/
-#define	OOR2	(17<<OPSHIFT)	/* | pt. 2	fwd to OOR2 or O_CH	*/
-#define	O_CH	(18<<OPSHIFT)	/* end choice	back to OOR1		*/
-#define	OBOW	(19<<OPSHIFT)	/* begin word	-			*/
-#define	OEOW	(20<<OPSHIFT)	/* end word	-			*/
-#define	OCHAR2	(21<<OPSHIFT)	/* character	(ch1<<8) | ch2		*/
-#define	OCHAR3	(22<<OPSHIFT)	/* character	(ch1<<16)|(ch2<<8)|ch3	*/
+#define	OEND	(1L<<OPSHIFT)	/* endmarker	-			*/
+#define	OCHAR	(2L<<OPSHIFT)	/* character	unsigned char		*/
+#define	OBOL	(3L<<OPSHIFT)	/* left anchor	-			*/
+#define	OEOL	(4L<<OPSHIFT)	/* right anchor	-			*/
+#define	OANY	(5L<<OPSHIFT)	/* .		-			*/
+#define	OANYOF	(6L<<OPSHIFT)	/* [...]	set number		*/
+#define	OBACK_	(7L<<OPSHIFT)	/* begin \d	paren number		*/
+#define	O_BACK	(8L<<OPSHIFT)	/* end \d	paren number		*/
+#define	OPLUS_	(9L<<OPSHIFT)	/* + prefix	fwd to suffix		*/
+#define	O_PLUS	(10L<<OPSHIFT)	/* + suffix	back to prefix		*/
+#define	OQUEST_	(11L<<OPSHIFT)	/* ? prefix	fwd to suffix		*/
+#define	O_QUEST	(12L<<OPSHIFT)	/* ? suffix	back to prefix		*/
+#define	OLPAREN	(13L<<OPSHIFT)	/* (		fwd to )		*/
+#define	ORPAREN	(14L<<OPSHIFT)	/* )		back to (		*/
+#define	OCH_	(15L<<OPSHIFT)	/* begin choice	fwd to OOR2		*/
+#define	OOR1	(16L<<OPSHIFT)	/* | pt. 1	back to OOR1 or OCH_	*/
+#define	OOR2	(17L<<OPSHIFT)	/* | pt. 2	fwd to OOR2 or O_CH	*/
+#define	O_CH	(18L<<OPSHIFT)	/* end choice	back to OOR1		*/
+#define	OBOW	(19L<<OPSHIFT)	/* begin word	-			*/
+#define	OEOW	(20L<<OPSHIFT)	/* end word	-			*/
+#define	OCHAR2	(21L<<OPSHIFT)	/* character	(ch1<<8) | ch2		*/
+#define	OCHAR3	(22L<<OPSHIFT)	/* character	(ch1<<16)|(ch2<<8)|ch3	*/
 
 /*
  * Structure for [] character-set representation.  Character sets are