Thank you for taking the time to read this! We are no experts in Tor networking, but know our way around.
If anything described here makes sense to you and/or you have a suspicion on what is going on, any kind of feedback is extremely appreciated!
1. What's happening?
Our service was running fine and stable for a long time. Suddenly our service experienced light, then severe performance issues at first, then became completely unreachable.
This does not seem to be a "regular" DDoS attack. It seems to be a DoS attack, but not like anything known, as it seems to be an attack on the Tor daemon alone. All countermeasures we tried so far were not successful. There is no attack noticeable on the actual services running (http, ssh, ftp, mail, etc.) but only on the Tor connection itself.
Even tough we did a deep research and investigation, the event seems to be beyond our comprehension. Documentations about prior attacks on Tor Hidden Services are rare to find and don't make sense to what we are experiencing.
We refuse to believe that there is an attack scenario that is unknown and cannot be counteracted as it would render every single Tor Hidden Service vulnerable to it with adversaries being able to take any Tor Hidden Service offline at will within minutes for an indefinite time. The impact on the Tor Community would be beyond scope.
We hope somebody with the necessary insight and expertise can at least drop us a hint on what exactly is going on and point us in the right direction towards finding a solution to end this attack or reducing it's impact on our systems and to prevent such attacks in the future. This would help protect Tor Hidden Services around the world from future attacks like the one we are experiencing.
2. What is the setup?
- The system runs on a current and always up-to-date linux (amd64) server
- All traffic is routed through Tor via Tor transport on port 9040, DNS queries are resolved via Tor as well
- The Tor Hidden Service is v3
- Tor is version 0.4.7.8 running with Libevent 2.1.12-stable, OpenSSL 1.1.1n, Zlib 1.2.11, Liblzma 5.2.5, Libzstd 1.4.8 and Glibc 2.31 as libc
- Tor compiled with GCC version 10.2.1. Vanguards 0.3.1 is also installed
- Tor is running as a systemd service
- Vanguards is running as systemd service
torrc configuration:
CookieAuthentication 1
HashedControlPassword 16:<hash>
VirtualAddrNetworkIPv4 10.192.0.0/10
AutomapHostsOnResolve 1
TransPort 127.0.0.1:9040
DNSPort 127.0.0.1:53
HiddenServiceDir /var/lib/tor/hidden_service
HiddenServicePort 80 127.0.0.1:80
HiddenServiceVersion 3
HiddenServiceAllowUnknownPorts 0
3. What are the symptoms?
- Less than a minute after starting Tor with the Hidden Service enabled the CPU gets to 100%, memory seems unaffected
- Used bandwidth increases around 100kb/s
- The Hidden Service descriptor is unreachable
- The system can still resolve clearnet and Tor addresses
- The system can still connect to outbound services (ex. curl to a website)
- The system gets flooded with incoming tcp packets on the loopback interface
- When Tor is restarted, the flood ends for a few seconds, then starts again
- When Tor is started without the Hidden Service enabled there seems to be no problem
- When Tor is started with a second Hidden Service enabled, both Hidden Service Descriptors remain unreachable:
Tor Browser on Primary HS:
Onionsite Has Disconnected
The most likely cause is that the onionsite is offline. Contact the onionsite administrator.
Details: 0xF2 — Introduction failed, which means that the descriptor was found but the service is no longer connected to the introduction point. It is likely that the service has changed its descriptor or that it is not running.
or
The connection has timed out
The server at (attacked hidden service descriptor).onion is taking too long to respond.
The site could be temporarily unavailable or too busy. Try again in a few moments.
Tor Browser on Seconday HS:
Unable to connect
Firefox can’t establish a connection to the server at (secondary hidden service descriptor).onion
The site could be temporarily unavailable or too busy. Try again in a few moments.
- When Tor is started with a second Hidden Service enabled which is protected by OnionAuthentication, the primary Hidden Service Descriptor remains unreachable,
the secondary (protected) Hidden Service Descriptor is reachable.
- When Tor is started with only the second Hidden Service enabled (with or without OnionAuthentication) there seems to be no problem
- When Tor is started with the primary Hidden Service protected by OnionAuthentication, the Hidden Service Descriptor is reachable
- When attacked, these entries appear in the tor log file:
Aug 24 19:42:18.000 [notice] Bootstrapped 100% (done): Done
Aug 24 19:42:34.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 388 buildtimes.
Aug 24 19:42:39.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 240 buildtimes.
Aug 24 19:42:53.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 489 buildtimes.
Aug 24 19:43:11.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 659 buildtimes.
...
Aug 24 19:46:09.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:46:09.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:46:09.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 114 buildtimes.
Aug 24 19:46:15.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 125 buildtimes.
...
Aug 24 19:47:08.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:10.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:10.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:13.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:14.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 495 buildtimes.
Aug 24 19:47:18.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 124 buildtimes.
Aug 24 19:47:21.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:23.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
...
Aug 24 19:47:55.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 1000 buildtimes.
Aug 24 19:47:59.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 117 buildtimes.
...
Aug 24 19:52:43.000 [notice] Strange value for circuit build time: 121581msec. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:52:43.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 120000ms after 18 timeouts and 57 buildtimes.
Aug 24 19:52:53.000 [notice] Interrupt: exiting cleanly.
4. What was tried to resolve the issue?
- We set up a completely new server from scratch running only basic OS as well as tor and vanguards in standard configuration to exclude the possibility of a mis-configuration on our affected server. As soon as tor was started with the attacked descriptor the exact same things are happening.
- We tried to split the load on our server via OnionBalance in this setup:
Server1: Runs Onionbalance for primary Hidden Service Descriptor
Server2/3/4: Runs the Hidden Service on new and different Hidden Service Descriptors
- This does not bring the original Hidden Service Descriptor back to life
- The service descriptors on servers 2/3/4 are reachable when opened directly, but not via the now balanced primary descriptor
- The CPU on servers 2/3/4 go to 100% as long as the attack takes place, while the CPU on server 1 (balancer) remains normal
- Adding more backend servers to the balancer did not make a difference either
- We added these directives to the hidden service block in torrc and tried various settings on them:
HiddenServiceEnableIntroDoSDefense 1
HiddenServiceEnableIntroDoSBurstPerSec <tested with various values>
HiddenServiceEnableIntroDoSRatePerSec <tested with various values>
This reduced CPU load significantly on servers 2/3/4, but the balanced service descriptor remains unreachable.
- We tried changing various settings in vanguards.conf, with no success.
- We tried to identify the attacking tcp packets and have them blocked via iptables, with no success.
Our expertise is not sufficient to draw ideas of what exctely is happening by inspecting the contents of said tcp packets which look like this:
19:45:27.839934 IP (tos 0x0, ttl 64, id 35746, offset 0, flags [DF], proto TCP (6), length 4100)
127.0.0.1.9051 > 127.0.0.1.46712: Flags [P.], cksum 0x0df9 (incorrect -> 0xe713), seq 1543428574:1543432622, ack 1711981309, win 512, options [nop,nop,TS val 2971851406 ecr 2971851369], length 4048
E.....@[email protected]........#[.x[...f
.............
."...".i650 CIRC 9802 EXTENDED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7B46F20449D6F25150E189428B62E1E3BA5848A9~galtlandeu,$BF93594384A02DE7689C4FD821E2638DA2CD4792~labaliseridicule BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.060324
650 CIRC 9802 BUILT $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7B46F20449D6F25150E189428B62E1E3BA5848A9~galtlandeu,$BF93594384A02DE7689C4FD821E2638DA2CD4792~labaliseridicule BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.060324
650 CIRC_MINOR 9802 PURPOSE_CHANGED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7B46F20449D6F25150E189428B62E1E3BA5848A9~galtlandeu,$BF93594384A02DE7689C4FD821E2638DA2CD4792~labaliseridicule BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_JOINED REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.060324 OLD_PURPOSE=HS_SERVICE_REND OLD_HS_STATE=HSSR_CONNECTING
650 CIRC 9818 EXTENDED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7A319C431F38CB30A0BC0C49144369A611920725~BahnhufPowah2,$8587A1B4CCD0700F164CCD588F79743C74FE8700~mev4PLicebeer16b BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.493699
650 CIRC 9818 BUILT $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7A319C431F38CB30A0BC0C49144369A611920725~BahnhufPowah2,$8587A1B4CCD0700F164CCD588F79743C74FE8700~mev4PLicebeer16b BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.493699
650 CIRC_MINOR 9818 PURPOSE_CHANGED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7A319C431F38CB30A0BC0C49144369A611920725~BahnhufPowah2,$8587A1B4CCD0700F164CCD588F79743C74FE8700~mev4PLicebeer16b BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_JOINED REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.493699 OLD_PURPOSE=HS_SERVICE_REND OLD_HS_STATE=HSSR_CONNECTING
650 CIRC 9997 EXTENDED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$8D896C8B367813030591A00DB7E7722EF6C4C23C~Luxembourg,$FF353F5D011E69ECDA10A57B46D06BC7B3FEB196~fuego,$347253D1D5246CB1C4CF8088C6982FE77CF7AB9C~ph3x,$E84F41FA1D1FA303FD7A99A35E50ACEF4269868C~Quetzalcoatl BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:25.100429
650 CIRC 9997 BUILT $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$8D896C8B367813030591A00DB7E7722EF6C4C23C~Luxembourg,$FF353F5D011E69ECDA10A57B46D06BC7B3FEB196~fuego,$347253D1D5246CB1C4CF8088C6982FE77CF7AB9C~ph3x,$E84F41FA1D1FA303FD7A99A35E50ACEF4269868C~Quetzalcoatl BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:25.10
This is with Tor running on a single server. When balanced, the |---------(attacked hidden service descriptor)---------| is replaced by the backends service descriptors of server 2/3/4.
We can make a larger tcpdump snipplet available, if needed.
5. Conclusions
We think that an adversary has the ability to "interrupt" a Hidden Service Descriptor (and therefore the Hidden Service itself) by flooding the Tor daemon with uncountable tcp packets, requesting to build circuits. This is what causes the CPU load and ultimately renders the Hidden Service Descriptor unusable.
Can anyone confirm this?
Since directives like the said HiddenServiceEnableIntroDoSDefense, HiddenServiceEnableIntroDoSBurstPerSec and HiddenServiceEnableIntroDoSRatePerSec seem to be meant do defend against this sort of attacks, just as vanguards should too, we cannot explain why those remain ineffective. Maybe some very specialized settings of those values are necessary to make them effective. Unfortunately these directives (as well as the settings in vanguards.config) are only described in a vague way.
Does anyone know, how these have to be set correctly to be effective?
At this point we exhaused all referneces on tor and vanguards configuration we could find online.
Again, any help or information on this issue would be greatly appreciated! We do not believe that there is no solution to this.
If anything described here makes sense to you and/or you have a suspicion on what is going on, any kind of feedback is extremely appreciated!
1. What's happening?
Our service was running fine and stable for a long time. Suddenly our service experienced light, then severe performance issues at first, then became completely unreachable.
This does not seem to be a "regular" DDoS attack. It seems to be a DoS attack, but not like anything known, as it seems to be an attack on the Tor daemon alone. All countermeasures we tried so far were not successful. There is no attack noticeable on the actual services running (http, ssh, ftp, mail, etc.) but only on the Tor connection itself.
Even tough we did a deep research and investigation, the event seems to be beyond our comprehension. Documentations about prior attacks on Tor Hidden Services are rare to find and don't make sense to what we are experiencing.
We refuse to believe that there is an attack scenario that is unknown and cannot be counteracted as it would render every single Tor Hidden Service vulnerable to it with adversaries being able to take any Tor Hidden Service offline at will within minutes for an indefinite time. The impact on the Tor Community would be beyond scope.
We hope somebody with the necessary insight and expertise can at least drop us a hint on what exactly is going on and point us in the right direction towards finding a solution to end this attack or reducing it's impact on our systems and to prevent such attacks in the future. This would help protect Tor Hidden Services around the world from future attacks like the one we are experiencing.
2. What is the setup?
- The system runs on a current and always up-to-date linux (amd64) server
- All traffic is routed through Tor via Tor transport on port 9040, DNS queries are resolved via Tor as well
- The Tor Hidden Service is v3
- Tor is version 0.4.7.8 running with Libevent 2.1.12-stable, OpenSSL 1.1.1n, Zlib 1.2.11, Liblzma 5.2.5, Libzstd 1.4.8 and Glibc 2.31 as libc
- Tor compiled with GCC version 10.2.1. Vanguards 0.3.1 is also installed
- Tor is running as a systemd service
- Vanguards is running as systemd service
torrc configuration:
CookieAuthentication 1
HashedControlPassword 16:<hash>
VirtualAddrNetworkIPv4 10.192.0.0/10
AutomapHostsOnResolve 1
TransPort 127.0.0.1:9040
DNSPort 127.0.0.1:53
HiddenServiceDir /var/lib/tor/hidden_service
HiddenServicePort 80 127.0.0.1:80
HiddenServiceVersion 3
HiddenServiceAllowUnknownPorts 0
3. What are the symptoms?
- Less than a minute after starting Tor with the Hidden Service enabled the CPU gets to 100%, memory seems unaffected
- Used bandwidth increases around 100kb/s
- The Hidden Service descriptor is unreachable
- The system can still resolve clearnet and Tor addresses
- The system can still connect to outbound services (ex. curl to a website)
- The system gets flooded with incoming tcp packets on the loopback interface
- When Tor is restarted, the flood ends for a few seconds, then starts again
- When Tor is started without the Hidden Service enabled there seems to be no problem
- When Tor is started with a second Hidden Service enabled, both Hidden Service Descriptors remain unreachable:
Tor Browser on Primary HS:
Onionsite Has Disconnected
The most likely cause is that the onionsite is offline. Contact the onionsite administrator.
Details: 0xF2 — Introduction failed, which means that the descriptor was found but the service is no longer connected to the introduction point. It is likely that the service has changed its descriptor or that it is not running.
or
The connection has timed out
The server at (attacked hidden service descriptor).onion is taking too long to respond.
The site could be temporarily unavailable or too busy. Try again in a few moments.
Tor Browser on Seconday HS:
Unable to connect
Firefox can’t establish a connection to the server at (secondary hidden service descriptor).onion
The site could be temporarily unavailable or too busy. Try again in a few moments.
- When Tor is started with a second Hidden Service enabled which is protected by OnionAuthentication, the primary Hidden Service Descriptor remains unreachable,
the secondary (protected) Hidden Service Descriptor is reachable.
- When Tor is started with only the second Hidden Service enabled (with or without OnionAuthentication) there seems to be no problem
- When Tor is started with the primary Hidden Service protected by OnionAuthentication, the Hidden Service Descriptor is reachable
- When attacked, these entries appear in the tor log file:
Aug 24 19:42:18.000 [notice] Bootstrapped 100% (done): Done
Aug 24 19:42:34.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 388 buildtimes.
Aug 24 19:42:39.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 240 buildtimes.
Aug 24 19:42:53.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 489 buildtimes.
Aug 24 19:43:11.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 659 buildtimes.
...
Aug 24 19:46:09.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:46:09.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:46:09.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 114 buildtimes.
Aug 24 19:46:15.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 125 buildtimes.
...
Aug 24 19:47:08.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:10.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:10.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:13.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:14.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 495 buildtimes.
Aug 24 19:47:18.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 124 buildtimes.
Aug 24 19:47:21.000 [notice] Extremely large value for circuit build timeout: 122s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:47:23.000 [notice] Extremely large value for circuit build timeout: 123s. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
...
Aug 24 19:47:55.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 1000 buildtimes.
Aug 24 19:47:59.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 117 buildtimes.
...
Aug 24 19:52:43.000 [notice] Strange value for circuit build time: 121581msec. Assuming clock jump. Purpose 14 (Measuring circuit timeout)
Aug 24 19:52:43.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 120000ms after 18 timeouts and 57 buildtimes.
Aug 24 19:52:53.000 [notice] Interrupt: exiting cleanly.
4. What was tried to resolve the issue?
- We set up a completely new server from scratch running only basic OS as well as tor and vanguards in standard configuration to exclude the possibility of a mis-configuration on our affected server. As soon as tor was started with the attacked descriptor the exact same things are happening.
- We tried to split the load on our server via OnionBalance in this setup:
Server1: Runs Onionbalance for primary Hidden Service Descriptor
Server2/3/4: Runs the Hidden Service on new and different Hidden Service Descriptors
- This does not bring the original Hidden Service Descriptor back to life
- The service descriptors on servers 2/3/4 are reachable when opened directly, but not via the now balanced primary descriptor
- The CPU on servers 2/3/4 go to 100% as long as the attack takes place, while the CPU on server 1 (balancer) remains normal
- Adding more backend servers to the balancer did not make a difference either
- We added these directives to the hidden service block in torrc and tried various settings on them:
HiddenServiceEnableIntroDoSDefense 1
HiddenServiceEnableIntroDoSBurstPerSec <tested with various values>
HiddenServiceEnableIntroDoSRatePerSec <tested with various values>
This reduced CPU load significantly on servers 2/3/4, but the balanced service descriptor remains unreachable.
- We tried changing various settings in vanguards.conf, with no success.
- We tried to identify the attacking tcp packets and have them blocked via iptables, with no success.
Our expertise is not sufficient to draw ideas of what exctely is happening by inspecting the contents of said tcp packets which look like this:
19:45:27.839934 IP (tos 0x0, ttl 64, id 35746, offset 0, flags [DF], proto TCP (6), length 4100)
127.0.0.1.9051 > 127.0.0.1.46712: Flags [P.], cksum 0x0df9 (incorrect -> 0xe713), seq 1543428574:1543432622, ack 1711981309, win 512, options [nop,nop,TS val 2971851406 ecr 2971851369], length 4048
E.....@[email protected]........#[.x[...f
.............
."...".i650 CIRC 9802 EXTENDED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7B46F20449D6F25150E189428B62E1E3BA5848A9~galtlandeu,$BF93594384A02DE7689C4FD821E2638DA2CD4792~labaliseridicule BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.060324
650 CIRC 9802 BUILT $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7B46F20449D6F25150E189428B62E1E3BA5848A9~galtlandeu,$BF93594384A02DE7689C4FD821E2638DA2CD4792~labaliseridicule BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.060324
650 CIRC_MINOR 9802 PURPOSE_CHANGED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7B46F20449D6F25150E189428B62E1E3BA5848A9~galtlandeu,$BF93594384A02DE7689C4FD821E2638DA2CD4792~labaliseridicule BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_JOINED REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.060324 OLD_PURPOSE=HS_SERVICE_REND OLD_HS_STATE=HSSR_CONNECTING
650 CIRC 9818 EXTENDED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7A319C431F38CB30A0BC0C49144369A611920725~BahnhufPowah2,$8587A1B4CCD0700F164CCD588F79743C74FE8700~mev4PLicebeer16b BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.493699
650 CIRC 9818 BUILT $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7A319C431F38CB30A0BC0C49144369A611920725~BahnhufPowah2,$8587A1B4CCD0700F164CCD588F79743C74FE8700~mev4PLicebeer16b BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.493699
650 CIRC_MINOR 9818 PURPOSE_CHANGED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$CED577F091DCB15AD8C87FBD452A51EA9E60BFC2~strayWires,$CC8B218ED3615827A5DCF008FC62598DEF533B4F~mikrogravitation02,$7A319C431F38CB30A0BC0C49144369A611920725~BahnhufPowah2,$8587A1B4CCD0700F164CCD588F79743C74FE8700~mev4PLicebeer16b BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_JOINED REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:22.493699 OLD_PURPOSE=HS_SERVICE_REND OLD_HS_STATE=HSSR_CONNECTING
650 CIRC 9997 EXTENDED $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$8D896C8B367813030591A00DB7E7722EF6C4C23C~Luxembourg,$FF353F5D011E69ECDA10A57B46D06BC7B3FEB196~fuego,$347253D1D5246CB1C4CF8088C6982FE77CF7AB9C~ph3x,$E84F41FA1D1FA303FD7A99A35E50ACEF4269868C~Quetzalcoatl BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:25.100429
650 CIRC 9997 BUILT $2BCA0A8B5759DBD764BF9FA5D1B3AEE9D74D2B68~Waeswynn,$8D896C8B367813030591A00DB7E7722EF6C4C23C~Luxembourg,$FF353F5D011E69ECDA10A57B46D06BC7B3FEB196~fuego,$347253D1D5246CB1C4CF8088C6982FE77CF7AB9C~ph3x,$E84F41FA1D1FA303FD7A99A35E50ACEF4269868C~Quetzalcoatl BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_SERVICE_REND HS_STATE=HSSR_CONNECTING REND_QUERY=|---------(attacked hidden service descriptor)---------| TIME_CREATED=2022-08-24T19:45:25.10
This is with Tor running on a single server. When balanced, the |---------(attacked hidden service descriptor)---------| is replaced by the backends service descriptors of server 2/3/4.
We can make a larger tcpdump snipplet available, if needed.
5. Conclusions
We think that an adversary has the ability to "interrupt" a Hidden Service Descriptor (and therefore the Hidden Service itself) by flooding the Tor daemon with uncountable tcp packets, requesting to build circuits. This is what causes the CPU load and ultimately renders the Hidden Service Descriptor unusable.
Can anyone confirm this?
Since directives like the said HiddenServiceEnableIntroDoSDefense, HiddenServiceEnableIntroDoSBurstPerSec and HiddenServiceEnableIntroDoSRatePerSec seem to be meant do defend against this sort of attacks, just as vanguards should too, we cannot explain why those remain ineffective. Maybe some very specialized settings of those values are necessary to make them effective. Unfortunately these directives (as well as the settings in vanguards.config) are only described in a vague way.
Does anyone know, how these have to be set correctly to be effective?
At this point we exhaused all referneces on tor and vanguards configuration we could find online.
Again, any help or information on this issue would be greatly appreciated! We do not believe that there is no solution to this.
Last edited:
