Can't create instances of Apple private classes in segment 's'?

[jerrykrinock]

Great book – I think I’m eventually going to have to read all the way through to Chapter 27. I’m only up to Chapter 6 so far, but have come to a stumbling block.

I need my macOS app to do some stuff which only Safari can do. I found what I need in, more or less, eleven classes in /System/Library/PrivateFrameworks/Safari.framework. I created an XPC helper tool in my app, added to it an abridged version of that framework’s header generated by class-dump, and linked my tool to it. Result: I am able to freely use two of these classes, and my code works as expected. For the other nine classes, although their instance methods work once I have an instance, I get a linker error when I attempt to create an instance using a convenience class initializer, alloc-init or new. For example, in the following demo code, I send new to all eleven classes. I have commented the two which //WORK:

BookmarkChange* bc = [BookmarkChange new];
BookmarkChangeTracker* bct = [BookmarkChangeTracker new];
BookmarksController* bmcon = [BookmarksController new];  // WORKS!!
BookmarksUndoController* buc = [BookmarksUndoController new];
WebBookmark* wb = [WebBookmark new];
WebBookmarkExporter* wbe = [WebBookmarkExporter new];
WebBookmarkGroup* wbgroup = [WebBookmarkGroup new];  // WORKS!!
WebBookmarkImporter* wbi = [WebBookmarkImporter new];
WebBookmarkLeaf* wbleaf = [WebBookmarkLeaf new];
WebBookmarkList* wblist = [WebBookmarkList new];
WebBookmarkProxy* wbp = [WebBookmarkProxy new];

For the other nine, I get linking errors:

Undefined symbols for architecture x86_64:
  "_OBJC_CLASS_$_BookmarkChange", referenced from:
  objc-class-ref in Demo.o
  "_OBJC_CLASS_$_BookmarkChangeTracker", referenced from:
  objc-class-ref in Demo.o
  // etc., 7 more times

Hmmm… why are these symbol undefined? I ran nm on that private framework, and with some grep, got the following results for those 11 classes. Note that I have passed the argument -U, which means Don’t display undefined symbols.

ACA80003:~ jk$ nm -U /System/Library/PrivateFrameworks/Safari.framework/Safari | grep _OBJC_CLASS | grep -E -e "BookmarkChange|BookmarksController|BookmarksUn|WebBookmark"
0000000000a16bb8 s _OBJC_CLASS_$_BookmarkChange
0000000000a16c08 s _OBJC_CLASS_$_BookmarkChangeTracker
0000000000a1a0b0 S _OBJC_CLASS_$_BookmarksController
0000000000a17108 s _OBJC_CLASS_$_BookmarksUndoController
0000000000a1a178 s _OBJC_CLASS_$_WebBookmark
0000000000a229e0 s _OBJC_CLASS_$_WebBookmarkExporter
0000000000a1a1a0 S _OBJC_CLASS_$_WebBookmarkGroup
0000000000a22a30 s _OBJC_CLASS_$_WebBookmarkImporter
0000000000a1a1c8 s _OBJC_CLASS_$_WebBookmarkLeaf
0000000000a1a1f0 s _OBJC_CLASS_$_WebBookmarkList
0000000000a160c8 s _OBJC_CLASS_$_WebBookmarkProxy

All eleven appear, but, aha, the two classes that work have a uppercase S preceding their names, and the nine that fail have a lowercase s. The man page says that S indicates that this symbol is in a section other than those above. The “above” are: undefined, absolute, text, data, bss and common. Furthermore, man states that If the symbol is local (non-external), the symbol’s type is instead represented by the corresponding lowercase letter. That’s weird because I’d expect the external ones to be the ones that fail to link, but we got the opposite. The Linux man page for nm has a different explanation for S: The symbol is in an uninitialized data section for small objects. Sigh, none of this is yet helpful to me.

Hmmm… are these nine possibly defined in a different module? Following pg. 29 and elsewhere in the book, I attached to the real Safari.app and did a image lookup on each of the 11 classes. Here is the result of the first such:

(lldb) image lookup -Av -s "BookmarkChange"
2 symbols match 'BookmarkChange' in /System/Library/PrivateFrameworks/Safari.framework/Versions/A/Safari:
    Address: Safari[0x0000000000a16bb8] (Safari.__DATA.__objc_data + 14040)
    Summary: (void *)0x001dffff9cb75be1
     Module: file = "/System/Library/PrivateFrameworks/Safari.framework/Versions/A/Safari", arch = "x86_64"
     Symbol: id = {0x0000cab7}, range = [0x00007fff9cb75bb8-0x00007fff9cb75be0), name="BookmarkChange", mangled="OBJC_CLASS_$_BookmarkChange"
    Address: Safari[0x0000000000a16be0] (Safari.__DATA.__objc_data + 14080)
    Summary: (void *)0x001dffff9deda0f1
     Module: file = "/System/Library/PrivateFrameworks/Safari.framework/Versions/A/Safari", arch = "x86_64"
     Symbol: id = {0x0000cab8}, range = [0x00007fff9cb75be0-0x00007fff9cb75c08), name="BookmarkChange", mangled="OBJC_METACLASS_$_BookmarkChange"

It looks OK to me (except I wonder why there are two matching symbols). The other ten results look exactly the same except for the symbol names and addresses. In particular, all eleven are said to be in module /System/Library/PrivateFrameworks/Safari.framework, which is the one that my helper tool is linking to.

Does anyone know what might be going on? I’m worried that the lowercase s means that this is some new Apple secret segment which is thus far impenetrable.

And thank you to anyone who made it to the end of this too-long post.

[lolgrep]

Hey @jerrykrinock, thank you for the kind words. I’ll answer the how while kinda dodging around the question of why. At the moment, I don’t know under what compiler situations Objective-C classes will be internal (s) vs external (S).

However, I do have advice on working with private frameworks and classes. Typically, I’ll use class-dump or even better, my own tool dclass (available here). Instead of generating a header file for a class, I’ll generate a protocol and cast the instance so it conforms to the generated protocol. From there, I’d use NSClassFromString to go after the private class of interest.

So let’s take a specific example, after you have loaded the Safari framework (either at runtime w/ dlopen or linking via compiler options), I’ll search for a specific class. Let’s go after WebBookmarkList

After Safari is loaded in memory, using LLDB…
(lldb) dclass -P WebBookmarkList

This will generate a file in /tmp called DS_WebBookmarkListProtocol.h
The contents of this file will contain the following:

#import <Foundation/Foundation.h>

@class NSArray;


@protocol DS_WebBookmarkListProtocol <NSObject>

@property (nonatomic) void * numberOfDescendants;
@property () void * automaticallyOpensInTabs;
@property (nonatomic) NSArray* folderAndLeafChildren;
@property (nonatomic) NSArray* leafChildren;
@property (nonatomic, getter=isReadingListFolder) void * readingListFolder;

+(id)smallImageForBookmarkList;
+(id)_test_allDescendentsOfBookmarkList:(id)arg0 ;
-(id)copyWithZone:(void *)arg0 ;
-(id)icon;
-(void *)numberOfChildren;
-(void *)canOpenInTabs;
-(void)getFolderAndLeafChildrenWithCompletionHandler:(id)arg0 ;
-(void)getDescendantsPassingTest:(id)arg0 completionHandler:(id)arg1 ;
-(void)getLeafChildrenWithCompletionHandler:(id)arg0 ;
-(void *)isUserVisiblyEqualToBookmark:(id)arg0 ;
-(void)clearDAVSyncState;
-(void)enumerateChildrenUsingBlock:(id)arg0 ;
-(void)setAutomaticallyOpensInTabs:(void *)arg0 changeWasInteractive:(void *)arg1 ;
-(NSInteger)bookmarkType;
-(id)childAtIndex:(void *)arg0 ;
-(void)removeChild:(id)arg0 ;
-(void)insertChild:(id)arg0 atIndex:(void *)arg1 ;
-(void *)indexOfChild:(id)arg0 ;
-(void)enumerateChildrenWithOptions:(void *)arg0 usingBlock:(id)arg1 ;
-(id)iconURLString;
-(id)firstChildWithURL:(id)arg0 ;
-(void *)mergeAttributesFromBookmark:(id)arg0 ;
-(void *)numberOfChildrenMatchingBookmark:(id)arg0 ;
-(id)_children;
-(void)enumerateLeafDescendantsUsingBlock:(id)arg0 ;
-(id)initWithIdentifier:(id)arg0 UUID:(id)arg1 group:(id)arg2 ;
-(void)_updateStateHash:(void *)arg0 ;
-(id)_mutableDictionaryRepresentation;
-(id)initFromDictionaryRepresentation:(id)arg0 topLevelOnly:(void *)arg1 withGroup:(id)arg2 ;
-(void)updateByCopyingSyncDataFromBookmark:(id)arg0 withChildBookmarksByUUID:(id)arg1 ;
-(void)clearAllChangesRecursively;
-(void *)updateByCopyingAttributesFromBookmark:(id)arg0 withExistingBookmarksByUUID:(id)arg1 ;
-(id)_childrenIncludingFolders:(void *)arg0 ;
-(void)_appendDescendantsPassingTest:(id)arg0 toArray:(id)arg1 ;
-(void)_enumerateLeafDescendantsOnChildrenAccessQueueUsingBlock:(id)arg0 ;
-(id)test_allDescendents;

@end

Now, I’ll import this header into the project and now I can do the following:

  id <DS_WebBookmarkListProtocol> a= [NSClassFromString(@"WebBookmarkList") new];
  
  NSLog(@"%@", a);

This protocol approach has the benefit of not having to deal with linker issues, not accidentally redefining an Objective-C class, and Xcode helps me autocomplete the private methods given from the protocol

Now, if I wanted to go after and use a custom initialization method for WebBookmarkList (i.e. like the initWithIdentifier:UUID:group: selector), I’d do the following:

  id <DS_WebBookmarkListProtocol> a= [(id<DS_WebBookmarkListProtocol>)[NSClassFromString(@"WebBookmarkList") alloc] initWithIdentifier:@"hello world" UUID:nil group:nil];
  
  NSLog(@"%@", a);

Have fun!

[jerrykrinock]

Hello, @lolgrep.

The short version is THANK YOU … NSClassFromString is the tip I needed. I’m almost done reverse-engineering that private API now, and am about ready to start writing actual code.

Longer version…

Regarding s versus S classes, after looking at some disassembly with otool, I’m convinced that methods in both s and S classes are definitely implemented in that private framework. So this s versus S thing is apparently one of the linker issues you referred to.

Regarding your approach to use protocol rather than interface, I agree it is safer, although somewhat harder on the eyes. And I think I’m only going to use NSClassFromString for those few class methods where it is needed. Since linking is done prior to me shipping, if something links, I don’t think that adding NSClassFromString is going to make it any more future-proof. Xcode’s autocomplete works with either protocol or interface to private API – understandable but surreal!

Since using NSClassFromString last night, I’ve made a lot of progress by guessing how APIs worked . I would love to have fun as you suggested, but, honestly, if I can avoid learning how to probe Safari with DTrace … you know, if I can get this job done without additional education, sadly, I must do that and move on to other tasks.

But I am sleeping better now, knowing that your Chapter 27 is there if I need it. Thanks, again!