It's been closer to 20 years since I last read a complete program in x86 assembly, so this is quite fun to look at.
I'm somehow disappointed (quite unreasonably, of course) that the code uses plain old zero-terminated C strings instead of something more exotic. One of the fun things about assembly is that you get to reinvent basic language features on the fly -- calling conventions, data layout, strings, everything.
AFAIK Linux doesn't use zero-terminated strings anywhere in its syscalls, or at least not in those like write, where you pass a size alongside the buffer.
As a general principle, bugs are reduced when you use representations that don't allow inconsistent representations. Unless you have an overriding reason, it's best to use a data representation that doesn't allow non-canonical representations (if such exists).
If you null-terminate a length-prefixed string, what if there's a null in the middle of the string?
(1) Allow inconsistency, and go with the length prefix in case of inconsistency. You could allow null bytes in the middle, treating it as a normal length-prefixed string, but then why do you null-terminate the string? (Is it so that you can still pass the string to functions that will choke on embedded nulls? Why would you do that?) This is just asking for kernel bugs.
(2) Allow inconsistency and go with the position of the first null byte in the case of inconsistency. If the length prefix is inconsistent with the position of the first null byte, you could go with the position of the first null byte, but then why even have the length prefix?
(3) Disallow inconsistency. You could disallow embedded nulls, but then the length prefix is just there as a place to cache strlen calls? If you're defining a syscall interface and requiring the length and first null to be consistent, then you need to run strlen anyway in order to sanity check what userspace gave you... why not simplify the external interface to just be either null-terminated or length-prefixed?
I should have specified a little more, perhaps. Don't think of it as a null-terminated + length-prefixed string. It's effectively a purely length-prefixed string. There just happens to always be a null one byte after the end of the string.
The length prefix is the only thing you use, ordinarily. The only time the null comes into play is if you've already had a bug.
... but one that doesn't terminate execution, but instead hides your bugs. In most use cases, I'd prefer to find my bugs in the majority of cases, rather than to hide the bugs except for corner cases.
Well, for a HTTP server, I don't have a specific idea... But in general, the fun part would be trying to come up with string representations that are optimized for the particular application.
The original 1984 Elite computer game is famous for its huge galaxy full of planets. Each of them had individual names and descriptions such as "Lave is most famous for its vast rain forests and the Laveian tree grub."
Yet those strings were never stored as plain strings. The game had to run in 32kB of memory, so almost all strings were stored in a tokenized form and expanded using a pseudo-random number generator:
That article shows how the planet description strings were stored and reconstructed on the fly. The base representation for the aforementioned description of planet Lave was only a handful of bytes: "\x8F is \x97"
So I think Elite is a pretty good example of an application written in assembly that didn't have anything like a generic string type.
That is awfully cool stuff, thank you for that ! but to be honest none of that is really specific to assembly. You can do the same thing in C/C++/Ada/Rust/pick your favorite systems language, or even in a higher level language if you don't care so much about data representation.
I would argue this has nothing to do with ASM, one could achieve the same compression with any programming language. The bottom of the linked article has a nice, few lines of Python.
As others have said, have a length field in with the string. This also has advantages other than making buffer overflows a lot harder, such as making string copying faster and easier. For example, let's have a skeletonized view of a normal string copy routine in assembly (disclaimer: my assembly is rusty, so this may not be completely right. void where prohibited):
push rax ; save our registers
push rdi
push rsi
mov rsi, location ; get the pointer to the right place
mov rdi, destination
beginning:
mov rax, [rsi] ; copy contents to the register so we can compare
test rax, rax ; compare our source to itself, if it's zero, it'll set a flag
jz done ; we're done
movsb ; copy the byte, increment the rsi and rdi registers
jmp beginning
done:
pop rsi ; restore our registers
pop rdi
pop rax
In contrast, with a length parameter, we can do
push rcx ; using a different register here
push rdi
push rsi
mov rsi, location ; same as before
mov rcx, length ; moving our length into the counter register
mov rdi, destination
cld ; okay our first change. Clearing the direction flag so the copy
; goes from the first byte to the end
rep movsb ; it'll repeat cx times the movsb command, and then carry on
pop rsi ; restoring our registers
pop rdi
pop rcx
Having the length of strings means you can have much more concise code. It makes loops easier, makes your code cleaner, and in some environments, gives a speed boost.
Handling strings with (ptr, length) also means that some string copies can be entirely avoided. Text can be chopped, shared and extended by adjusting the two parameters, while the underlying storage remains untouched.
e.g. a really basic example for a web server would be splitting up a URL into a path and query string: both strings can use the underlying URL without any copying.
Lots to add to that if you want: sanity check the length against external buffer limit; move word/dword/mmx128; consider alignment; not use deprecated movs instruction.
In fact the best, alignment-sensitive solution I've never seen written. It would load 2 large words, shift them to target alignment if needed, store, load, repeat. This would guarantee aligned fetch/store and still do whole-bus operations.
I've waited to see an instruction to do this in any machine ever (why do we have to hand-code this kind of thing, when the processor chip KNOWS the best way to get it done?) I've waited 20 years.
You mean vectorizing memcmp/memcpy to work word-at-a-time instead of byte-at-a-time? Google does this when compiling their binaries, and I think Facebook does too. I thought the latter had open-sourced it with Folly but couldn't find a code pointer. I've heard rumors that LLVM can sometimes vectorize them to use SIMD instructions when available, too.
It's hard to do this safely for strcpy/strcmp because you might read past the end of the buffer when trying to test against a null terminator. memcmp/memcpy and length-prefixed blocks let you use a Duff's-Device-like construct to test only the last word byte-by-byte.
In a paged system, reading past the end of the string, but no further than the last whole aligned word containing the null terminator, will never page fault. So that can still work.
I've waited to see an instruction to do this in any machine ever (why do we have to hand-code this kind of thing, when the processor chip KNOWS the best way to get it done?) I've waited 20 years.
REP STOS (memset) has also gotten the same boost throughout the generations of x86, and if the trend continues I'd expect REP CMPS and LODS to get the same treatment. These string instructions are tiny (1-2 bytes) and yet very powerful; their greatest advantage is that they don't take up the astoundingly large amount of space in the icache that some extremely micro-optimised routines (i.e. ridiculous amounts of loop unrolling) do.
Turbo/Borland Pascal used a NUL-optional string format of length (byte IIRC) followed by data. [0]
Java IIRC also uses a length-oriented format for string constants in .class files. [1] It's been a while since I wrote a .java to MIPS asm compiler in C++ from scratch (don't ask).
This is because real-world strings may contain 0 to N NULs and escaping them is too much of a PITA for serialized formats, so it's easier and common to do things like TYPE LENGTH DATA de/serialization. For modern, efficient binary de/ser, check out binc and msgpack [2,3].
UTF-8 is not well suited for a general purpose string implementation because it is a variable length encoding and therefore addressing a character becomes a linear time operation. UTF-16 would probably be a better choice in most cases.
UTF-16 is also a variable length encoding and addressing a character is a linear time operation. Then again, even UTF-32 can have composed characters, such as ¨a separately forming ä.
True but UTF-16 captures really a very large share of actually used code points. UTF-32 captures all of them but wastes at least 11 bit per code point, more right now because most of the code points are unassigned. It seems a good tradeoff for a lot of use cases and as you mentioned once operating on code points is no longer good enough you will have to face the issue regardless of the encoding.
UTF-8 consumes less space and has pretty much same trade-offs in terms of iteration. No worries about endian (UTF-16LE vs UTF-16BE). Almost all input text is in UTF-8 and so is almost all output text. Conversion back and forth to UTF-16 is just wasted CPU time.
I think even counting number of UTF-8 code points in a string is faster in UTF-8 than in UTF-16, if you're allowed to use SSE2/AVX2/AVX-512, because all UTF-8 sequences start with a byte that has highest bit 0, all other bytes in the sequence have highest bit 1.
So just SIMD vector compare [1] to find all "positive" bytes (highest bit == 0), which gives you a nice mask. Then move the mask to a general purpose register [2] and popcount [3] it. 16/32/64 (SSE2/AVX2/AVX-512) bytes processed at a time, no branches other than loop control branch.
You can use the same idea to quickly scan UTF-8 string to approximately right position to retrieve a given (random) code point index. Still O(n), but with 10-50x smaller constant factor. If that's not enough, you can simply pre-index every n-th code point (say, every 64/128/256th) in a separate array for larger UTF-8 strings. That gives you constant time random access.
Note: UTF-8 code points start either 0xxxxxxx or 11xxxxxx. Regardless of this the basic idea should work, just need to do two compares and to bitwise-or the masks. At the end, AVX-512 of course would use mask-register (k0-k7) and AVX2 would probably need to convert the mask in two parts, once for both 128-bit register halves.
Note 2: Thinking about it a bit more, I think it's enough to check if signed bytes are greater or equal than -64 (0xc0)! This covers bit patterns from 11000000 (-64) to 01111111 (127); all the sequences that can start a sequence. So no two compares and bitwise-or needed after all.
Tested my idea quickly. Initial messy (but correct) version finishes in 31% of time that version you linked [1] (cp_strlen_utf8) takes to run. I think I can still improve it quite a bit.
Only tested with long 30 MB strings.
Edit: Now at 26%. But it can still be improved more. Both benchmarks are with hot cache.
Edit 2: Well, 4x performance 32 bit, but compiling it 64-bit in VS2015RC manages to optimize cp_strlen_utf8 more, almost doubling performance. 45% then. Will try gcc 5, clang, etc. later. And it can still be optimized further.
Edit 3: Ended up at 35% (2.9x) execution time for 64-bit and 18% (5.5x) for 32-bit. My version is as fast in 32 and 64-bit, but cp_strlen_utf8 benefits quite a bit from 64-bit mode. Probably memory bandwidth limited at this point, but I didn't profile yet. In any case, it does utf-8 code point strlen at 16 GB/s at this point. CPU is i5-4430 CPU @ 3.00GHz, two memory channels @1600 MHz.
> and therefore addressing a character becomes a linear time operation. UTF-16 would probably be a better choice in most cases.
> but UTF-16 captures really a very large share of actually used code points.
That most characters[1] in use are a single code unit in UTF-16 is meaningless to code that needs to index[2] into a UTF-16 by code point (or grapheme): the only correct way to accomplish this in a typical UTF-16 string implementation is O(n).
[1]: I love emoji, and they are outside the BMP.
[2]: I think you'll find that most code does not need to index into a string. (Though languages that lack iterators on strings will make writing the code without indexing difficult.)
UTF-8, UTF-16 and UTF-32 are different encodings of the same character set, so i'm not sure what you mean in UTF-16 captures really a very large share of actually used code points.
next, your claim that UTF-8 is not well suited for a general purpose string implementation because it is a variable length encoding and therefore addressing a character becomes a linear time operation is incoherrent: UTF-16 is a variable length encoding just as well. come out and say that you want to be lazy and pretend surrogate pairs don't exist.
I used Rust and Swift languages a little bit and I must admit that I almost never had to use indexing for characters. Iterating and slicing is enough for most algorithms.
Actually today it might be hard to understand what indexing is, even if you store your string in UCS-32 encoding. There are graphical symbols that may occupy variable number of UCS-32 items. And they are used out there (e.g. flags).
I remember one program that had a string print subroutine that took zero args. It just used the return address from the stack to grab the nul-terminated string immediately following the JSR/CALL instruction. It then patched the return address on stack to return just after the nul.
Bad for storing data in .text, but still a neat hack that shaved whole tens of bytes off the program size.
I'm somehow disappointed (quite unreasonably, of course) that the code uses plain old zero-terminated C strings instead of something more exotic. One of the fun things about assembly is that you get to reinvent basic language features on the fly -- calling conventions, data layout, strings, everything.