Recent site activity

Mac OS X‎ > ‎

Bonjour Sleep Proxy

I've been trying to get Bonjour Sleep Proxy working off and on for a long time.  I'm starting to collect some details to help me when I'm back "on" again.


Wake on Demand/Wake on LAN/Wake on Wifi/Wake on Magic Packets

Apple refers to the basic feature with widely different names but usually mean the same thing.  I use the 4 names interchangeable on this page.

A good starting point is to make sure you can remotely wake up your machine using magic packets.  I initially tried this with OS X 10.6; which is a good thing as I may have been misled by 10.8's (mis)behavior.

I tried a few tools to remotely wake the computer by sending a Magic Packet. I had most success with tools that used UDP packets directed to Port 9.

My iMac is new enough that Wifi reports it supports Wake on Demand; as described by this Apple page. So I tested this by connecting to my local network using Wifi and disconnecting my Ethernet cable.  Then I ran pmset sleepnow from a terminal to put the iMac to sleep.

From another computer, I used some standard WOL tools to send the magic packet.  After a few seconds, I'd try to ssh to my iMac.  First try rarely woke up the computer.  It appears the wifi chip sleeps deep enough that it misses a lot of packets.  So I sent maybe 10 magic packets in a row and my computer would reliable wake up and I could ssh to it.

Next, I disabled Wifi and reconnected the Ethernet cable and repeated the test.  I could reliably wake up my iMac with only 1 magic packet sent.

One other random note.  I used the following python script from a Linux box to send a custom Ethernet frame (non-IP) using Ethertype 0x0842 packet since thats what Bonjour Sleep Proxy's send.  I was able to reliably wake up the iMac over ethernet but I've not tested Wifi.  This script only works on Linux as far as I know because it needs AF_PACKET.

#!/usr/bin/python
from socket import socket, AF_PACKET, SOCK_RAW

s = socket(AF_PACKET, SOCK_RAW)
s.bind(("wlan0", 0))


# Set this to your local MAC
src_addr = "\x66\x55\x44\x33\x22\x11"
# Set this to what you want to wake up
dst_addr = "\x11\x22\x33\x44\x55\x66"
ethertype = "\x08\x42"
payload = ("\xff"*6)+(dst_addr*16)
s.send(dst_addr+src_addr+ethertype+payload)

Also, I tried using ether-wake which is pre-installed on my RT-N66U router that has asuswrt-merlin firmware installed on it.  I believe, but I am not positive, that this also uses plain ethernet with 0x0842 since it doesn't request an IP address that UDP needs.  I was unable to wake the iMac with this tool from router.  I was able to wake it using the wol tool pre-installed which uses UDP packets.  This means I can at least wake up my Macs using the router's WOL web page since that is the tool it uses on backend

Wake on Demand and OS X 10.8

I upgraded from 10.6 to 10.8 and discovered that magic packets seemed to stop working.  After some googling, this seems a common problem for people starting with 10.7.

I would send the magic packet and couldn't ssh into my box.  If I manually woke up the computer, I could tell that it was actually being woken up but just didn't seem to be initializing the network stack and/or allowing ssh to accept connections.  I could tell it was woken up from syslog and timestamps:

$ syslog | grep "Wake reason"

May 21 20:37:36 my-imac kernel[0] <Debug>: Wake reason: GIGE (Network)


Since Wake on Demand over Wifi is treated as an aftertought in the OS, the message during that wake usually says "Wake reason: ?".

The issue seems to be related to the Power Nap feature but internally it seems to be called Dark Wake alot.  I'm not totally sure why Apple didn't think to have Wake on Demand fully bring up networking but perhaps one day they will fix that.  In mean time, you can disable the Dark Wake/Power Nap feature which doesn't change the way the box sleeps but simply how far awake it becomes.

Dark Wake is built into the kernel and you can modify its behavior with options passed to the kernel during bootup.  In the kernel shipped with 10.7, a darkwake flag was introduced.  It is a bit mask that was initialized to a value of 3.  You can see sample code from the open sourced kernel around that timeframe:


// gDarkWakeFlags
enum {
    kDarkWakeFlagHIDTickleEarly      = 0x01, // hid tickle before gfx suppression
    kDarkWakeFlagHIDTickleLate       = 0x02, // hid tickle after gfx suppression
    kDarkWakeFlagHIDTickleNone       = 0x03, // hid tickle is not posted
    kDarkWakeFlagHIDTickleMask       = 0x03,
    kDarkWakeFlagIgnoreDiskIOInDark  = 0x04, // ignore disk idle in DW
    kDarkWakeFlagIgnoreDiskIOAlways  = 0x08, // always ignore disk idle
    kDarkWakeFlagIgnoreDiskIOMask    = 0x0C,
    kDarkWakeFlagAlarmIsDark         = 0x0100
};

static uint32_t         gDarkWakeFlags = kDarkWakeFlagHIDTickleNone;

The common work around for 10.7 users is to edit /Library/Preferences/SystemCongfiguration/com.apple.Boot.plist and modify Kernel Flags to look like this:

<key>Kernel Flags</key>
<string>darkwake=0</string>

After a reboot, waking from magic packets should work same as it was in 10.6.

In 10.8 kernel, the default value for the bitmask changed to 11 (0xb).  You can also see this from open source kernel around that timeframe:

static uint32_t         gDarkWakeFlags = kDarkWakeFlagHIDTickleNone | kDarkWakeFlagIgnoreDiskIOAlways;

I don't know exactly what the new 0x08 does *but* if you clear that bit with darkwake=0 then although the computer will again wake from magic packets like in 10.6, it will no longer go to sleep on its own.  Clearing bits 1 and 2 (0x3); like in 10.7 fix; but leaving the new bit 5 (0x8) still set seems to get the desired behavior.

<key>Kernel Flags</key>
<strig>darkwake=8</string>

So now I have 10.8 back to where I can manually wake it up.

Which Interfaces support Bonjour Sleep Proxy?

Somewhere along this 10.6 to 10.8 time frame I got an Apple TV and learned that this device can perform the role of a Bonjour Sleep Proxy Server.  So I began to see if I could get that working but haven't had much success.

First step was to learn which Apple products can support Sleep Proxy on client side.  I found that the feature is controlled by mDNSResponder daemon.  You can run this command to have it dump out some needed info to /var/log/system.log:

sudo killall -INFO mDNSResponder
less /var/log/messages

Look for section at very end that lists Network Interfaces:

May 21 21:10:13 my-imac mDNSResponder[42]: ------ Network Interfaces ------
May 21 21:10:13 my-imac mDNSResponder[42]: xxxxxxxxxxxxxxxxx 4, Registered 0 xxxxxxxxxxxx, v4 en0 (xxxxxxxxxx) 11:22:33:44:55:66 00:00:00:00:00:00 Active v4 192.168.1.4   v6 ⊙ ⇆ ☀

The interesting part are the symbols.  If you see a sun, as above, it says that this Network Interface supports Wake on Demand and that its detected a Sleep Proxy on your network.  If its a sun but with clear center then it means this Network Interface supports Wake on Demand but no Sleep Proxy was seen on network.  If there is no sun symbol then it means mDNSResponder thinks your interface does not support Wake on Demand.

I've found that all Wifi interfaces claim Wake on Demand is not supported inside mDNSResponder; regardless of what your System Information window says or the fact that Apple swears it works on Wifi.  I've checked on an iMac and a Macbook Air.

I also found enabling addition logging from mDNSResponder is helpful in pinpointing where things are going wrong:

sudo killall -USR1 mDNSResponder
sudo syslog -c mDNSResponder en

Perform a forced sleep and then after waking up, search for the phrase WOMP (Wake on Magic Packet).

$ pmset sleepnow
$syslog | grep WOMP
May 21 20:37:47 my-iMac.local mDNSResponder[42] <Notice>: en0    xxxxxxxxxxxxxxxxx supports WOMP

If it thinks its not supported then it will say "no WOMP".

If your motivated enough, it is the NetWakeInterface() function that (mis)detects if the interface supports WOMP (Wake on Magic Packet).  You could hack it to see if this is "en0" or "en1" and force to supports regardless of kernel flags.  The following is piece of code that needs modification:

LogSPS("%-6s %#-14a %s WOMP", i->ifinfo.ifname, &i->ifinfo.ip, (ifr.ifr_wake_flags & IF_WAKE_ON_MAGIC_PACKET) ? "supports" : "no");
return((ifr.ifr_wake_flags & IF_WAKE_ON_MAGIC_PACKET) != 0);

Take care to not say ever device supports it.

That change alone is probably not enough since; as stated above; waking from Wifi usually takes several packet sends but it looks like Sleep Proxy Servers only send 2 packets.   It would be difficult to fix your Apple TV or Airport.  If you  have an Apple computer you always leave running, you can tell it to support being a Sleep  Proxy Server and hack up SendWakeup() to send multiple times.

For now, I'm sticking with my Ethernet port and disabling my Wifi ports.

Sleep Proxy Clients in NIC

Some newer Mac's support a mini-Sleep Proxy in the NIC so the computer doesn't need to wake up to refresh th advertised services.   If you enable above -USR1 logging option and sleep, you can grep for this support:

$ syslog | grep BeginSleepProcessing
May 21 19:22:28 my-imac mDNSResponder[42] <Notice>: BeginSleepProcessing: en0 Found Sleep Proxy Server

The above says my computer does not have that support in the NIC and it will communicate to Sleep Proxy Server on its own.  If that line instead said "using local proxy" then mDNSResponder will not be involved in maintaining the advertised services.

I mention this mostly in case I get a newer computer some day and see unexpected packets sent that do not match up with syslog.  Also, if you try to fix the Wifi not support WOMP, be sure and read up on "-UseInternalSleepProxy 0" option to mDNSResponder to disable use of NIC proxy.

Detecting Sleep Proxy Server Registration

When the Mac is going to sleep, it must first register with the Sleep Proxy server and tell it to be its proxy.  I found the following the easiest way to see if this registration is successful.  On a remote machine, ssh to the Mac to sleep.  While connected to Mac to sleep, from another terminal on remote machine run the arp command.  Verify that the MAC is in fact the correct MAC of Mac to sleep:

$ arp -a
my-imac (192.168.1.4) at 11:22:33:44:55:66 on en0 ifscope [ethernet]

Now on the ssh, run pmset sleepnow and then use ssh's "~." to quit without doing something that might wake computer up before sleeping.  Wait a few seconds and then run arp command again.  You should notice the MAC address change to the MAC of your Sleep Proxy server:

$ arp -a
my-imac (192.168.1.4) at 66:55:44:33:22:11 on en0 ifscope [ethernet]

If life is good, you can ssh back to Mac to sleep, the Sleep Proxy Server should send some magic packets, and it should wake up and complete ssh connection.

Seeing this arp table helps you understand how the Sleep Proxy Server has a hook to know when to wake up your Mac.  Since arp caches are cleared over time (usually 60 minutes) then it might also give some clues to why the sleep proxy behavior changes over time.

Sleep Proxy Server Interactions

To debug problems with a Sleep Proxy Server, its helpful to understand how they work.  A Sleep Proxy Server must advertise a _sleep-proxy._udp service.  You can detect this with the dns-sd command and Sleep Proxy Clients perform the same query right before going to sleep:

$ dns-sd -B _sleep-proxy._udp local
Browsing for _sleep-proxy._udp.local
DATE: ---Sat 25 May 2013---
15:49:36.127  ...STARTING...
Timestamp     A/R Flags if Domain                    Service Type              Instance Name
15:49:36.128  Add     2  4 local.                    _sleep-proxy._udp.        xx-xx-xx-xx.x Apple TV

The client will also need to know the Name and Port the Sleep Proxy Server is listening on for this service:

$ dsn-sd -L "xx-xx-xx-xx.x Apple TV" _sleep-prox._udp local
Lookup xx-xx-xx-xx.x Apple TV._sleep-proxy._udp.local
DATE: ---Tue 28 May 2013---
19:23:54.865  ...STARTING...
19:23:55.440  xx-xx-xx-xx\.x\032Apple\032TV._sleep-proxy._udp.local. can be reached at Apple-TV.local.:52956 (interface 4)

Before the client goes to sleep, it must register with the server.  It does this by sending a unicast Dynamic DNS Update which contains at a minimum a list of the TCP services (using SRV and TXT records) that should cause the computer to wake up, a Dynamic DNS Update Lease OPT record and a EDNS0 Owner OPT record directly to the server name and port from previous query.

The Dynamic DNS Update Lease OPT record seems to be the main trigger to cause the server to proxy for the client.  It contains a Time To Live value that the client must wake up and re-register with server before the server disables the proxy feature.  The EDNS0 Owner OPT record contain the client's MAC address to send in magic packet. The SRV  records are used as a list for port to monitor for TCP SYN messages; all other ports are ignored.  Each SRV record contains its own Time To Live value but its not clear to me if the the server will stop listening when they expire or use the OPT records TTL.  Once all services expire, the server will also stop responding to ARP requests for the client; effectively ending its proxy roll until the client re-registers.

Once a client sends this packet, the server will send out a gratuitous ARP packet with its own MAC address to clear out machine's ARP caches.  It will also begin to respond to ARP requests for client's IP address with its own MAC address; as long as one or more services still haven't timed out.  An ARP request alone will not cause a request to wake up the client.

If someone attempts to establish a TCP connect to one of the services, the packet will be sent to server because of the ARP response.  The server will detect this and send out a magic packet to wake the client but doesn't respond to the TCP connection request.  Once client awakes, it sends its own gratuitous ARP which reroutes the initial TCP connection packets to client and it responds to complete connection.

That the main job of the server.  It looks like the server will also be a proxy and respond to mDNS requests on behalf of the client.  In practice, all machines will have the client's services cached in their local mDNS cache since the client wakes up roughly every 2 hours and send out a multicast list of its services with a TTL of 1 hour and 15 minutes.  The last job of the service is to send out a magic packet when the TTL of the OPT record expires as a last ditch effort to get client to refresh its services; although in practice the Mac's wake themselves up before this magic packet is ever sent.  The Linux based SleepProxyClient script takes advantage of this though instead of using an RTC timer to wake themselves up.

Since Apple products do not send IP based magic packets, you can't log them with Linux iptables commands.  You can use ebtables logging though.  Here is what I use on my Linux based router to log magic packets to syslog:

ebtables -A FORWARD -p 0x842 --log-level warning --log-prefix "WOL"

If you have tcpdump install somewhere, here is example of logging unicast Dynamic DNS Updates, multicast DNS messages but only from sent from 192.168.1.4 (the sleeping Mac), and non-IP magic packets and save them to a file to be later parsed by wireshark.  Note: Port 53956 is based on response from above dns-sd -L command.  Its a value changes alot.

tcpdump -i eth1 -s 65535 'udp port 52956 or (udp port 5353 and src 192.168.1.4) or ether proto 0x0842' -w packet.pcap

Local Sleep Proxy Server

If you have a spare Mac; besides the one your trying to be a Sleep Proxy Client and wake from sleep; you can tell its mDNSResponder daemon to be a Sleep Proxy Server. This gives you the ability to enable logging and better debug your issues. This is enabled by editing /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist and -OfferSleepProxyService 10 to the start options.  Find the sections that looks similar to this and add the missing lines:

    <key>ProgramArguments</key>
    <array>

        <string>/usr/sbin/mDNSResponder</string>

        <string>-launchd</string>

        <string>-OfferSleepProxyService</string>

        <string>10</string>

    </array>

Be warned that you must go to System Preferences->Power Savings->Computer sleep option to Never.  If this option is set to let computer sleep then mDNSResponder will NOT become a Sleep Proxy.  The value of 10 is also important because it gives higher priority to this proxy compared to any other that may be on your network.

You must restart your computer for changes to take effect or you can issue this two commands to restart mDNSResponder without rebooting:

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl uoad /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

I've used this and the killall -USR1 to log timestamps of when server starts responding to ARP requests and when it stops responding to them.  Seems it stops responding to them at roughly 75 minutes after the last Dynamic DNS Update message was sent from Client.

Back To My Mac

Apple has an iCloud service called Back To My Mac which lets you connect to your computer from a remote location.  I believe its somewhat meant to work with sleeping Macs as well.  As best I understand, the sleeping computer's mDNSResponder must perodically wake to refresh its registered services with both any local Sleep Proxy Server and to the iCloud Back To My Mac Servers but that a local Sleep Proxy Server is always required to send the magic packets to wake the sleeping Mac.

To ssh from a command line from a remote location, you'll need to find the ID given to your iCloud account by running dns-sd command:

$ dns-sd -E
Looking for recommended registration domains:
DATE: ---Sat 25 May 2013---
16:06:47.578  ...STARTING...
Timestamp     Recommended Registration domain
16:06:47.579  Added     (More)               local
16:06:47.579  Added                          icloud.com
                                             - > btmm
                                             - - > members
                                             - - - > 12345678

The above tells you the iCloud address for this account is 1245678.members.btmm.icloud.com.  If you wanted to look for computer names registered with ssh service for that account you can run dns-sd -B _ssh 12345678.members.btmm.icloud.com.  If you know the name of your computer (convert spaces to dashes and drop any other special character) you can ssh to it with ssh username@my-imac.12345678.members.btmm.icloud.com.

If I look at my router's port forwarding table, I see that ports related to Back To My Mac are being registered multiple times and on NAT boxes you can only forward 1 port 1 time.  If your router only supports UPNP registration then I think only 1 computer can register to BTMM at a time.  If it uses NAT-PMP then that supports a way for clients to request a port # but a different one be returned.  When you look at output on your router from NAT-PMP registration you should see unrelated IP addresses and ports 4500 and 5353 mapped to unrelated ports on:

Destination     Proto. Port range  Redirect to     Local port
ALL             UDP    5353        192.168.1.4     5353      
ALL             UDP    4502        192.168.1.5     4500      
ALL             UDP    5355        192.168.1.5     5353        
ALL             UDP    4503        192.168.1.6     4500      
ALL             UDP    5356        192.168.1.6     5353  
    
I know its the miniupnp daemon that suports both UPNP and NAT-PMP so I'll watch its changes.

If your having issues with BTMM, look at your port forwards and verify that you computer in question is registered and has a non-conflicting port value on the WAN side.

Lose of Services During Sleep

Using both an Apple TV 2 and an Macbook Air as a Sleep Proxy Server, I notice that over time the sleeping Mac's services slowly stop showing up in Bonjour Browser for the local network.  When using the Apple TV 2, the services also disappear from Back To My Mac/iCloud network most the time.  Using Macbook Air, the BTMM network seems to keep advertising the services longer.  I can't quite nail down exact timing of when the services are lost.

I've seen reports that BTMM works more reliably for people in waking their Mac's up which kinda aligns with above.  But on the other hand, my experience tells me that its the sleep proxy that wakes up the Mac in all cases and once it loses knowledge of a service for sleeping Mac, it also stops sending Magic Packets to wake the Mac.

If your sleeping Mac doesn't wake up in a timely fashion, the services will expire and be removed.  You can use the following command to get a feel for what time intervals mDNSResponder is waking up your Mac:


$ syslog | grep lightweight

May 27 15:42:04 my-imac mDNSResponder[42] <Notice>: AllowSleepNow: Will request lightweight wakeup in 3217 seconds
May 27 16:36:08 my-imac mDNSResponder[42] <Notice>: AllowSleepNow: Will request lightweight wakeup in 6466 seconds
May 27 18:51:15 my-imac mDNSResponder[42] <Notice>: AllowSleepNow: Will request lightweight wakeup in 3328 seconds

From above log, you can tell its sometimes setting to wake up in about 1 hour and 45 minutes and other times in about 1 hour and you can tell the timestamps correspond roughly to those time periods.  mDNSResponder likes to wake up 10% early compared to Time To Live values its tracking.  Those values above roughly correspond with TTL of 75 minutes and 120 minutes.

So it seems my Mac is attempting to do the right thing and wake up occasional to refresh its advertised services.  Its a little suspicious to me that its using the 6466 values.  I've done some initial investigation into what a Sleep Proxy Client does (mostly by browsing this python script) when it goes to sleep to see how long the Sleep Proxy Server will wait before timing out the client's services.

The server not responding to ARP's after 75 minutes; which is same time period as Time To Live in Dynamic DNS Update for the services.   Here is a small snippet of decoding packet from wireshark:

My iMac._ssh.tcp.local: type TXT, class IN, cache flush
  Name: My iMac._ssh._tcp.local
  Type: TXT (Text strings)
  .000 0000 0000 0001 = Class: IN (0x0001)
  1... .... .... .... = Cache flush: True
  Time to live: 1 hour, 15 minutes
  Data length: 1
  Text:

Turns out the TTL of 120 minutes comes from the OPT record in the Dynamic DNS Update message that the sleeping client sends right before sleeping.  It also tracks multiple outstanding TTL's from DHCP and other items and wakes up after the shortest TTL.

Wireshark doesn't decode this OPT record but you can see it by looking at the data.  120 minutes is 7200 seconds which is 0x201c or 0x1c20 stored in little endian format:

00 02 00 04 00 00  1c 20 ...

On of the RFC's says that this OPT TTL is supposed to be the value used for overall Dynamic Update; regardless of the TTL of the TXT and SRV records.

Changing to Short TTL

As a test, I tried to shorten the TTL of the Dynamic DNS Update from 2 hours to 1 hour and 15 minutes.  I had no success in recompiling the mDNSResponder because it requires to may private headers to be installed that do not ship with OSX.  So instead I patched the binary.  The TTL is set in SendSPSRegistrationForOwner() with value DEFAULT_UPDATE_LEASE (7200).

You can run this command to disassemble the mDNSResponder binary:

otool -tV /usr/sbin/mDNSResponder | less

Search for _SendSPSRegistration: to find begining of this function and then search for 7200. It is the one that looks like this:


000000010003f380 movq -1504(%rbp), %rax
000000010003f387 movw $2, 4(%rax)
000000010003f38d movq -1504(%rbp), %rax
000000010003f394 movw $4, 6(%rax)
000000010003f39a movq -1504(%rbp), %rax
000000010003f3a1 movl $7200, 8(%rax)

So take note of the 0x3f3a1 value (ignore the 1 set up higher) and use your favorite binary editor to change the value. Here is one way:

xdd /usr/sbin/mDNSResponder ~/mDNSResponder.hex
vim /~/mDNSResponder.hex
/03f3a0 <-- search for 03f3a0. Round last digit to 0 always
[look for 201c and change to 1194. Save changes and exit]
xdd -r ~/mDNSResponder.hex ~/mDNSResponder

I replaced my local binary with this modified version and verified it now sends Dynamic DNS Update with new TTL and client always wakes up with in 1 hour.  I still have issues with services being lost.  So its not a real fix but does make it randomly work a little more often then it was.

Large Number of Services

I have a large number of services enable: ssh, sftp, VNC, File Sharing, iTunes audio streaming, and Home Sharing.  When the client sends this list to the server, there must be a limit on size of message.  RFC 1035 mentions something about maximum size of 512 bytes.  So it appears that the client has to send at least 2 messages during sleep registration and I verified this happens correctly using wireshark.  Normally, I see the services listed in the first packet and then some simple info about IP addresses of the client in the second packet.

As long as client is sleeping, the sleep proxy seems to work.  If the client wakes up because of the "lightweight" wake mentioned above, then it will resend a new Dynamic DNS Update message.  More often then not and this resend , it only sends 1 packet instead of 2 and that 1 packet is usually the one with limited info.

From that point on, the sleep proxy no longer knows about the now missing info and so will not respond to any services and wake up the Mac.

In parallel, the client is maintaining a list of services with iCloud's Back To My Mac.  With that unicast DNS server, it does not need to send as many services; namely it doesn't send the iTunes audio streaming (DAAP) or Home Sharing services.  This shrinks it enough that it can more often send in 1 message.  So sometimes you wan wake up your Mac by using the Back To My Mac address as apposed to the private LAN name.

Work around?  Turn off some of your shared services until its able to register with a single message.  I've been playing with this and so far its working its been working fine... although I'd really like to have those services re-enabled!


Comments