We have basically no cellular connectivity or internet access at our new home.
We will in the near future have a slightly better (but still slow) 4g connection thanks to our cell site booster, and DSL (about two miles from the DSLAM.)
I have invested a lot of time and energy in to offline first applications, but not in a while, so I'm re-investigating these things specifically as they relate to our new housing situation.
One tool that I have played with in the past is Kiwix.
Kiwix is an offline wikipedia reader, and it's pretty useful, especially when paired with Kiwix serve.
It seems like you should be able to abstract it in to a full fledged LAN to internet bridge, leveraging RSS support and maybe a couple of hacky hotsync jobs to download content when you have a network connection and then sync it with your LAN machine when you're on your LAN.
This doesn't work, though.
It has a few pretty severe limitations that relegate it to the world of Just Wikipedia, and not more generally useful for me, and I always forget about that until I try to build my own content for it.
Basically, it uses the file format from zim-wiki, a desktop wiki tool. It's really not great at serving content from multiple websites at one time (I can make it work, but it's needlessly complicated) and building content for it is an absolute nightmare.
So I'm looking for something better.
I want to cobble together a LAN internet in a box.
Something like the piratebox project, but that enables me to sync in content from outside my network.
I have some options.
Dat/Beaker always looks like it would be a solution to this problem, but it's node based and the last time I used it the command line functionality was limited. I'll re-investigate it.
More importantly Dat only solves the *sync*, not the capture or the serve.
@elb @ajroach42 Oh yes, I can jump in. So, start with #Syncthing. I love it. I explored some possibilities with it at https://changelog.complete.org/archives/10219-a-simple-delay-tolerant-offline-capable-mesh-network-with-syncthing-optional-nncp . Syncthing is serverless sync, ala Dropbox, but also distributed. One possible scenario: a device (laptop or phone) is the "carrier". Syncs locally at the house, and to someplace on the Internet at a coffee shop / in town / whatever.
@elb @ajroach42 4/ What you are really after is more general asynchronous communication. I have a whole blog series about this, including #NNCP and other tools: https://changelog.complete.org/archives/tag/asynchronous will give you all the posts in the series. Many of them are somewhat focused on backups, but should give you some good ideas for other things also. NNCP can use things like USB sticks, serial links, regular Internet connections, Syncthing, etc. as transport.
@elb @ajroach42 6/ You talked about accessing web pages offline. I've tried that but mostly don't really bother. It is fairly painful (you frequently want to click on a link you don't have). In some cases, for things like larger articles, it can make good sense. But you might want to look into something more like rss2email . Email is already asynchronous and there are lots of ways to get asyncrhonous email across. NNCP is one and documents this workflow at http://www.nncpgo.org/Feeds.html
@elb @ajroach42 7/ Offline email is two separate problems: sending and receiving. Sending can go across BSMTP (delivered "somehow" via NNCP, UUCP, Syncthing, etc). I talk about Exim with NNCP as part of my series here https://changelog.complete.org/archives/10165-asynchronous-email-exim-over-nncp-or-uucp and the NNCP docs go over the Postfix setup. For incoming, you can use OfflineIMAP or an offline-capable mail reader. Alternatively, forwarding to an account you can receive via NNCP/etc to a local mailstore.
@elb @ajroach42 9/ Finally the two best ways to improve your 4G signal are: 1) height, and 2) antenna. I got one of these https://smile.amazon.com/gp/product/B01NBSLNJ6 with a Nighthawk M1 awhile back. Tremendous difference. A booster can only boost what it can receive. A good antenna, mounted high, hardwired into the access point will almost certainly be better. That antenna has "gain", meaning it's directional, so figure out where your best towers are and point it at those.
@elb @ajroach42 10/ Also point-to-point wireless may help; if there's a good place you can get Internet and you have line-of-sight from your house, you may be able to work something out, even something surprisingly fast. For more challenging conditions, LoRA or XBee could work... but at 100Kbps or less. Not suitable for browsing but could work for email.
@elb @ajroach42 11/ Finally, don't understimate the utility of sshing to a VPS somewhere and reading email in text. My qualifications to anser: have lived in Internet-challenged areas for 20 years, frequently travel into no-Internet areas, have experience with modern communication over extremely low bandwidth links (1 to 100Kbps) including LoRA/XBee radio, AX.25 packet radio, and satellite.
@jgoerzen @elb I did some experimenting with LoRA and tried to do some work with XBee for keeping nodes of a distributed BBS in sync over multiple KM, but ultimately we just didn't have the mesh density, and I ended up building a solution that used a device I carried with me to rsync each location over wifi.
Using syncthing and a cellphone (and NNCP) seems like a more viable longterm solution.
@nev a USB drive on average weighs about 30 grams, according to the two sources I found that weighed USB drives.
Microcenter cells 256 GB flash drives. You could fit 3000 of them in a 90 KG payload.
So that's 768 terabytes.
A full sized trebuchet can launch a 90kg at roughly 70m/s or 156.586 MPH.
So you're sending 768 TB at 300 meters in 4.2 seconds for a transfer speed of 182.857143 terabytes TBps.
@nev Or, in more conventional terms, 1462857144 Mbps, or 1462857144000 Kbps or 1.462857144e+15 bits per second.
@ajroach42 @nev I thought 30g sounded like a /really/ heavy flashdrive, so I weighed the small jar of flashdrives on my desk. their weights: 10g, 9g, 4g, 8g, 16g. which would increase data throughput :) I can weigh some SD cards too I guess... nvm. microSD card doesn't make it up to 1g on my scales. 0.5g would be best guess. regular sized SD card is about 2 or 2.5g.
I think what I'm wanting in terms of capturing web pages is an interface where I provide, for example, a URL or a search term, and the next time a connected node has an internet connection, that web page and every page it links to directly, and every page each of those links to directly is captured for LAN transfer at a later date.
@djsundog @ajroach42 Interestingly, Sergey just (literally within the past few days) added "multicast" support to #NNCP http://www.nncpgo.org/Multicast.html . The Internet-connected machine could send the downloaded site to the LAN "area". Upon arriving at the LAN NNCP gateway, it would be copied both to the PC and to the indexer. Wouldn't require any extra scarce bandwidth, and also wouldn't require the indexer to be up in order for the data to reach the PC.
"I appreciate SDF but it's a general-purpose server and the name doesn't make it obvious that it's about art." - Eugen Rochko