My TTFB has been really bad since I switched to the godaddy Managed VPS.
https://glaspak.com/test.htm - That is a simple html file that just contains "Hello World"... My TTFB for that is .244 seconds. I ran a TTFB on a random image from another website and I get back a 0.054 second ttfb... Makes no sense why my little html file would be 4 times slower than an image (granted it is another web server not hosted by godaddy).
You have a major misunderstanding on how networking works, but we wont go into that.
Simply put, you can not compare two complete data files to their speeds. Depending on modules enabled with your web-server and configurations set by the ISP & network administrator data being sent over the network can be compressed ( which is usually how data is sent ). So in your scenario, it could be that your small image file is actually sending less packets over the network as compared to the small html file which wouldn't have many byte patterns to compress.
If you want to run a reliable speed test do something like this on both servers using resources from the server you are testing:
Note, this is still not a pin point accurate result but it's close enough.
If you would like us to help you further post back the following information:
Apache Modules Enabled
Sites Available conf
Apache Error Logs
Modsec error log
Me and @glaspak stayed up all night reviewing his server and eventually only corrected SSL ciphers and Forward Secrecy with PCI compliance standards by comodo and trustwave. OP also updated kernel to address dirty cow exploit.
Tracing route to glaspak.com [220.127.116.11]
over a maximum of 30 hops:
1 2 ms 2 ms 2 ms . [10.0.0.1]
2 11 ms 13 ms 14 ms 10.197.96.1
3 12 ms 11 ms 11 ms ten0-9-0-1.orld71-car1.bhn.net [18.104.22.168]
please note that, i offer direct support that is in no way associated with GoDaddy's regular support channels; but, for immediate technical support you can contact me at: https://join.skype.com/t3cxhmNCspqu