Objective-C and Invalid UTF-8 Byte Sequences

I found out the following information the hard way. I hope it helps you in some way…

Objective-C does not handle invalid UTF-8 byte sequences gracefully.

NSString’s UTF8String method will simply return nil if it encounters any invalid chars (which could happen when deserializing a slightly corrupted file from disk, for example). The encoding conversion methods, even the ones that allow “lossy” conversions, will either return nil – or only return the first part of the string up to the invalid sequence. I guess “lossy” here only refers to valid characters.

I had a fiendish problem with one user, where a presumably corrupt char in the DB would cause UTF8String to return nil, and the whole thing fell over. I tried many alternatives to UTF8String like  NSData’s dataUsingEncoding:allowLossyConversion, and NSString’s getBytes:maxLength:usedLength:encoding:options:range:remainingRange.  The former simply returned nil too, the latter would give me the first part of the string.

What was worse, is that [stringToCheck canBeConvertedToEncoding:NSUTF8StringEncoding] would actually return YES (but then not convert).

My conclusion is that all these methods, and everywhere that “lossy” is referenced is designed for converting valid strings from one encoding to another. What I had was an invalid NSString, which these methods are not designed for, even canBeConvertedToEncoding:NSUTF8StringEncoding.

Third Party Solutions Are Hard to Come By

I searched outside the Objective-C/iOS SDK, and found a few leads. One was this solution which uses the Omni framework. I didn’t try it, as I didn’t want to add a massive dependancy for such a simple, rarely used function. It seems that people commonly use GNU’s iconv for the task, e.g. in Ruby and PHP, but I have no idea if I can even compile that for iOS, not to mention the whole LGPL issue.

My Solution to Fix Corrupt NSStrings

In the end, my solution was this: for strings that fail the UTF8String conversion (so only in 0.01% of the time), get every single UTF16 char from the NSString, and make it’s own NSString. Then, run UTF8String on just that single-char string to test if it worked, and ignore those that fail (actually, I replace them with the replacement char: ‘�’) . It’s not pretty, it’s not fast – but it works. And since this is a very rare case, generally the only performance hit you get is the overhead of the UTF8String test.

The only way I know how to test for sure that you can convert to UTF8 is to actually do the conversion via NSString’s UTF8String. It’s heavy, but it’s accurate…

Here’s the method:

/* UTF8 fixup methods, by William Denniss, http://williamdeniss.com/ */

/*
 * Convenience method to do the check and validation in one.
 */
+ (NSString*) makeValidUTF8:(NSString*) stringToCheck
{
	if (![Util isValidUTF8:stringToCheck])
	{
		return [Util removeInvalidCharsFromString:stringToCheck];
	}
	else
	{
		return stringToCheck;
	}
}

/*
 * Returns true if the string can be converted to UTF8
 */
+ (NSString*) isValidUTF8:(NSString*) stringToCheck
{
	return ([stringToCheck UTF8String] != nil);
}

/*
 * Removes invalid UTF8 chars from the NSString
 * This method is slow, so only run it on strings that fail the +Util::isValidUTF8 check.
 */
+ (NSString*) removeInvalidCharsFromString:(NSString*) stringToCheck
{
	NSMutableString* fixedUp = [[[NSMutableString alloc] initWithCapacity:[stringToCheck length]] autorelease];

	// iterates all characters of the string to check
	for (NSUInteger i = 0; i < [stringToCheck length]; i++)
	{
		// gets the character as a one-char string
		unichar character = [stringToCheck characterAtIndex:i];
		NSString* charString = [[NSString alloc] initWithCharacters:&character length:1];

		// converts it individually to UTF8, testing for errors
		if ([charString UTF8String] == nil)
		{
			NSLog(@"Invalid UTF-8 sequence encountered at position %lu. Code: %hu (%X). Replacing with \ufffd", (unsigned long) i, character, character);
			[fixedUp appendString:@"\ufffd"];
		}
		else
		{
			[fixedUp appendString:charString];
		}
		[charString release];
	}

	NSLog(@"Util:makeValidUTF8 WARNING: string was NOT valid utf-8.  Orig length %d, fixed length %d", [stringToCheck length], [fixedUp length]);

	//NSAssert([fixedUp UTF8String] != nil, @"still nil");

	return fixedUp;
}

Incidentally, the character that was in my NSString which caused UTF8String to fail and caused all my grief was U+D843

iOS Untrusted Server Certificate

Had a problem recently with iOS 3.0 where it did not trust my server’s SSL certificate.

Newer versions do, so iOS 3.0 must simply not have the root certificates.

There are two ways to solve this: one is to blindly accept all certificates. That’s bad – as you lose the identify protection that SSL provides (and subject your users to man-in-the-middle attacks). It’s also harder, as you have to apply this to *every* connection (and if you use a UIWebView that may not be possible). But if you really want, the info is here.

The better approach is to include the root certificates of the certificate authority (no need for your actual certificate) in your App. This just makes the iPhone trust your CA, and will even work if you re-issue the cert in the future with the same CA (no need to update the app).

To trust a certificate, simply add this method:

// ref: http://stackoverflow.com/questions/1746005/importing-an-ssl-cert-under-the-iphone-sdk
+ (void) trustCert:(NSString*)filename ext:(NSString*)ext
{
	// NB. certificate must be in DER format (SecCertificateCreateWithData will return nil if not)
	// To convert, on the command line type: openssl x509 -in your_cert_name.cer -out UTN-your_cert_name.der -outform DER

	OSStatus            err;
	NSString *          path;
	NSData *            data;
	SecCertificateRef   cert;

	path = [[NSBundle mainBundle] pathForResource:filename ofType:ext];
	assert(path != nil);

	data = [NSData dataWithContentsOfFile:path];
	assert(data != nil);

	cert = SecCertificateCreateWithData(NULL, (CFDataRef) data);
	assert(cert != NULL);

	err = SecItemAdd(
					 (CFDictionaryRef) [NSDictionary dictionaryWithObjectsAndKeys:
										(id) kSecClassCertificate,  kSecClass,
										cert,                       kSecValueRef,
										nil
										],
					 NULL
					 );
	assert(err == noErr || err == errSecDuplicateItem);

	CFRelease(cert);
}

I got that method from here, but added a check to see if the cert was already imported (and not assert on that case).

If you don’t have your certificates on hand, you will need to export all the certificates in the chain of trust. I did this from Firefox using the view certificate -> export feature. Export all certs in the tree, except your server’s one itself (this one is not needed). For me, this meant 3 certificates.

Make sure your cert is in DER format. You can convert the typical ASCII
CER format (such as the ones exported through Firefox) into DER in this way on the command line:

openssl x509 -in your_cert_name.cer -out UTN-your_cert_name.der -outform DER

Add your .der file to your project (all targets).

Then, just call this line before you use the cert:

[YourClass trustCert:@"your_cert_name" ext:@"der"];

For me, I had to trust all 3 certificates in the chain. The only cert I didn’t need to bundle was the one for my server itself.

WARNING: due to the way this code works, you only have to call it *once* and it keeps the cert in your app’s keychain (I think). Even if you delete the app, the cert remains there. So take care when testing your code. I think if you change the bundle identifier for your app, it would be like a fresh install. Remember, deleting the app will not delete the cert from that app’s keychain, so this can fool you into thinking the code is no longer needed (or something).

Importantly – this code only solves your cert issues from WITHIN your App. To solve them from Mobile Safari (these are two, mutually exclusive solutions), you need to create a mobile config file containing all your root certificates, and get the user to install it. You can do that with the iPhone Configuration Utility, create the config and send users to that URL on their iPhones.

Ability to Downgrade iPhones

There are many legitimate reasons to downgrade iPhones. As a user, perhaps you just don’t like the new OS.

As a developer, I have a need to test my App on all OS versions my users may reasonably use. I even have a spare iPhone I use just for testing. Alas, Apple only allow the very latest OS to be installed, even if you are a developer.

Apple also don’t ship old OS versions on the iPhone Simulator anymore, so that can’t be used to verify that you’re not calling any new methods either.

For a recent release, this meant I actually could not test it properly. I did the best I could by using the iPad 3.2 * simulator* (a very poor man’s substitute), crossed my fingers, and released. It was a nail-biting day…

Not all users upgrade immediately. Upgrading requires iTunes, yet normal day to day usage of an iPhone does not (you can do everything, buy/upgrade apps, buy music, etc). Actually I believe this is a pretty common use-case, especially for a key audience of mine: travellers. In fact, with my App, at least 10% of my active users (ones using the export server) are not yet on iOS4! That also means 10% of my potential customers. No way am I going to ignore them.

A bug has been reported to Apple on this issue.

To summarise, there are many legitimate reasons why people are on old OS’s, and why (especially developers) have a need to downgrade to old OS’s.

Fortunately there is a solution. And it doesn’t require doing anything illicit to your iPhone either.

Download The Firmware Umbrella. Plug in your iPhone to USB, and press a button.

What the Firmware Umbrella does is cache the iTunes server OK response to your firmware installation request, allowing you to fake this response in the future (to allow the downgrade). Only catch: you can only cache responses for the latest firmware, so do it today!

Important TIP: tick “Advanced Options” and change the “Request From” option to “Apple” (Cydia is for jailbreaking). Optionally, you can do both.

Caching the response is easy. Using it a bit harder – but do yourself a favour, cache the response now, and if you need it in the future, you’ll thank yourself.

Why there won’t be a GPS Log HD

When you port your App to the iPad, one of the first choices you get is whether to create a separate product, or a fat (i.e. “universal”) binary.

It seems that many folks on the App store, including several apps in the GPS/travel space have gone done the separate-product route.

Perhaps I understand why they did this. After all, it is not trivial to port an App to the iPad (basically your UI needs a complete re-think). For some apps, they have gone so far as to offer a totally new UI, using real life metaphors like “pages” and “books”.

I still think it’s a bad idea. Why? For two main reasons:
1) customers don’t like paying for the same product twice
2) how many people will actually buy both versions anyway?

Yes the iPad version costs you money to make. BUT, it will hopefully get you more customers. New customers who buy it for their iPad.

Historically I know of several products that used to ship with multiple targets. Warcraft 3 is one, and Adobe’s suite is another, both supported both Mac and Windows. Many steam games now support Mac as well (and don’t require re-purchasing).

So my theory is most users won’t buy the App twice. Some will buy it for the iPhone, some will buy it for the iPad. If you have the one binary to support both, a few will run it on both, and find that useful. If you sell it separately, I doubt most people would buy the counterpart, simply because they would use one device more than the other. Furthermore, users that have already bought the iPhone version get that installed to their iPad anyway, which may be enough.

So rather than trying to milk your existing customers to pay for the port, think of it this way: You get new customers (those who have never used your app, and want to use your App on the iPad), you add value to your product (by allowing use of both devices) which may give you an edge. And you’re doing the right thing (ask yourself, do you like paying for the same thing twice?).

I think so few customers will buy the app TWICE, that you actually de-value your product. I suspect that the number of additional people who buy your App because it supports both, will outweigh the number of people who would have bought a second copy, and that both categories of users represent only a small percentage anyway.

And please, don’t try to claim “but the iPad version is different”. I don’t care if it has a fancy book UI, a re-arranged layout, or up-res’d textures. If it performs the same function, it’s the same App.

GPS Log for iPad. Coming soon. Free for existing users.

Will

Hardcore iPhone Memory Debugging

NSZombies is great for finding references to deallocated objects (a common crash issue). But it is less useful when trying to debug a crash in autorelease, especially if the object in question is very common.

For that, you can enable malloc logging (in the simulator) and really trace where the object in question was allocated.

As stated in this excellent post, with NSZombieEnabled, MallocStackLogging and MallocStackLoggingNoCompact enabled, you can get the PID of the simulator from your debug console, and then run shell malloc_history from the debugger to get the stack trace.

More great articles:
iPhone Memory Management & Debugging
Debugging Autorelease (not iPhone specific)
More on this topic
not directly related, but a favourite on the topic of objc_msgSend bugs

Android Marketplace is Very Limited

I’ve been learning more about Android recently. I like what I see. Seems like cool hardware, some cool Apps, and a good philosophy behind the whole thing (open source OS, “democratic” app approval process).

However as far as making money is concerned, I’m really not convinced this is the place to do it.

I’m not sure just how much is being bought and sold on the App store, but there appears to be a really big deficiency. And that is you can only sell your apps to 13 countries. Not only that, but the rate at which countries are added seems very slow (no additional ones in over a year).

Here is a bit of a deal breaker for Australian developers, they cannot legally sell Android Apps.

Compared to Apple, which can sell to many countries. As one person puts it “Apple is running out of map” when trying to find new countries to sell in. They recently launched support for Botswana, amongst others.

There is a big discussion here

Interestingly anyway can view how much money an App has made (as a range). This is public information! And is used as a shit-filter (like Apple failed to do with star-ratings). As a user, that is cool, I’m interested to see what Apps have done well. As a developer this is terrible! It means a rival developer is able to make a very accurate cost-benefit analysis of whether copying my App would be profitable!! Not so good…

My thoughts for now are: Android is a fine place to create a freely distributed app (which you possibly make money off another way, a.l.a drop-box, or via ads), but as for paid-Apps, it seems that Apple is not only king, but it is a much bigger king with heaps more moolah.

iPad Simulator crashes if a UIDatePicker is in a UIPopoverController

Just had one of those afternoons, where you waste 2 hours trying to debug some really strange crash in fairly simple code only to find that the API is at fault.

I have a UIDatePicker which I present in a (custom) UIAlertView used on the iPhone.   On the iPad it makes sense to use a UIPopoverController.

However on use, it would crash about 20% of the time on the first showing, and 100% of the time on the second with various errors like:

*** -[_UIPickerViewSelectionBar lastClickRow]: unrecognized selector sent to instance 0x5570c80

and

*** -[_UIOnePartImageView lastClickRow]: unrecognized selector sent to instance 0x5576b70

and

*** -[NSCFString lastClickRow]: unrecognized selector sent to instance 0x6081770

Basically it’s just sending “lastClickRow” to junk…
And now… what cost me 2 hours but you get for free (hey why not check out GPS Log ;)

The workaround:  Set the UIViewController’s .view (to the view containing the UIDatePicker) AFTER you call presentPopoverFromRect on the UIPopoverController.

No visual glitches with the workaround.

Thanks A  ._.

I could file a bug report… but I can’t stand that the iPhone bug reports are hidden, i.e. I have no idea if 100s of others haven’t already filed the same one and I may just waste my time.  Already kinda regretting the waste of time on this issue.

Porting to iPad

Some useful resources for the task:

• iPad Programming Guide
• Thanks to Ray Wenderlitch for his excellent iPad porting tutorial
• Info from Apple on how to configure the universal binary:

Follow these steps to build a universal application that will run on both iPad and iPhone:

• Set the Base SDK build setting (in the Architectures section) to iPhone SDK 3.2.
• Set the iPhone OS Deployment Target build setting to iPhone OS 3.1.3 or earlier.
• Set the Targeted Device Family build option to iPhone/iPad.
• Make sure that your Architectures build setting uses both armv6 and armv7
• Set the Active SDK to iPhone Device 3.2, select your Distribution configuration, build (select the Build button) your application, and submit it for App review.

• Weak-linking the UIKit Framework
• setting the frame of a UIPopoverView

Auto resize the Default.png when in a call & tethering

The iPhone can automatically resize the Default.png to take into account the status bar, even when the status bar is double in size.

It even crops out a bit in the middle so your top and bottom toolbars still line up!

To take advantage of this, make your Default.png 320×460, not 320×480.  Simple :)

Thanks to samwize for the tip.

Making FBDialog.h Modal

UPDATE: I submitted a patch for this issue to the latest version of the Facebook iOS SDK, and it was recently approved.

FBConnect (now Facebook iOS SDK) for iPhone is a cool way to interact your App with Facebook. The FBDialog.h class is also a pretty cool way to display a HTML page to the user in a modal way.

Yeah it’s open source too (Apache 2.0) so no problem with appropriating it for other uses :D

However it’s not totally modal. You can still interact with the underlying view on the edges.  This is especially bad on the iPad where the dialog is much smaller than the screen.

Fixing this fortunately is easy:

Under – (void)show, replace:

[window addSubview:self];

with:

_modalView.frame = window.frame;
[_modalView addSubview:self];
[window addSubview:_modalView];

And in – (void)postDismissCleanup, replace

[self removeFromSuperview];

with

[_modalView removeFromSuperview];

Just add a member UIView* _modalView, init and release in the classes init and dealloc respectively :)

Easy.

Thanks @facebook for the useful code!