wgs

wgs, 10 months ago

You don’t need to access a .onion instance to use Tor. You can simply perform your day-to-day web usage through Tor directly.

On your phone, you can even use Tor natively with most of your apps.

wgs, 10 months ago (edited 10 months ago)

I’ve just started digging into it myself ! Here’s my current setup (I’ll see how it scales in the long term):

• syslog on every host
• Telegraf collects and parse logs
• InfluxDB stores everything
• Grafana for dashboards

I run OpenBSD on all my servers, and configure all the services to log via syslog.

Then I configuré syslog to send only those I care about (https, DNS, …) to a central telegraf instance, using the syslog protocol (RFC3164).

On this collector, telegraf gets all these logs and parse them using custom grok patterns I’m currently building, to make sense out of every log line it receives. The parsed logs are in turns stored in Influxdb, running on the same host.

I then use Grafana to query InfluxDB and create dashboards out of these logs. Grafana can also display the logs “as-is” so you can search through them (it’s not ideal though as you simply search by regex from the full message, so it’s on par with grep at least).

This setup is fairly new and seem to work very well. Telegraf is also very low on resource usage for now. I’ll have to continue adding grok patterns and send more application logs to it to see how it handles the load. I do have a few questions still unanswered for now, but time will tell:

Q: Should I first collect via a central syslog before sending to telegraf ?
This would let syslog archive all logs in plain text, rotate and compress them. I would also only have a single host to configure for sending logs to telegraf. However this would eat up space, and could hide the original sending hostname for each log. I might try that someday.

Q: Should I run telegraf on each host ?
This would distribute the load of the grok parsing amongst all hosts, and then all telegraf processes will send directly to the central one for collection, or even directly into influxdb. I would also benefit from telegraf being install on each host to collect more data (CPU, network stats, …). However it makes the configuration more complex to handle.

Q: What is a good retention period ?
For now, influxDB doesn’t expire any data, as I don’t have much yet. In the long run, I should probably delete old data, but it’s hard to tell what is “old” in my case.

Q: Do I need an interface to read logs ?
I use this setup mostly for graphs, as grafana can make sense of fields like “http_verb”, “http_code” and such. However, it is much more practical for me to dig into the logs right on the server, in /var/log. Having an interface like chronograf or graylog seems practical, but I feel like it’s overdoing it.

Bonus:unbound dashboard

wgs, 10 months ago

You’ll want to check this out: tumfatig.net/…/ads-blocking-with-openbsd-unbound8…

That’s the post I took inspiration from for this setup. It does use collectd and custom awk scripts for log ingestion though, where I simply use telegraf.

wgs, 10 months ago

It’s instant as well in my case, but I don’t have a huge amount of logs yet. I’m still figuring out this whole setup and what are it’s strength and weaknesses.

I’m using influxdb 1.8 though (which is old), because that’s the version shipped with openbsd repos. It crashes fairly often when you perform “illegal” operations, which is annoying. Like, the DELETE FROM command only lets you use the time field in the WHERE clause. Using any other field would crash the DB. I might recompile it from scratch at some point because it lacks too many features from upstream. But for now, it does a decent job, and is really easy to setup (this was the killer feature for me).

wgs, 10 months ago

Also works as PLC, how useful !

wgs, 11 months ago

And followed its white rabbit.

wgs, 11 months ago

You must be fun at parties

wgs, 11 months ago

Do not use Dendrite for multi-user setups if you plan to run bridges. Contacts handled by the bridges are visible by the whole server, which means that it leaks many information on your contacts (names, phone number, …). I’m also not sure that multi-user puppeting is supported with dendrite.

I would advise you to run Synapse because of that.

wgs, 11 months ago

I store and query them using influxdb. I checked Loki but apparently it’s main feature is that it store the message as a single field, this not parsing the log at all. I didn’t know about Promtail. Is it better suited than influxdb for my usecase ?

wgs, 11 months ago

It does help thank you ;)

I’ve found that you can use custom grok patterns to parse logs just as grayling extractors do. I’m still trying to figure it out, but so far I could start parsing logs using a [[processor.parser]] block. I’ll document my findings when I get it working as I want it.

wgs, 11 months ago

I found how to parse and tokenize logs withing telegraf. One must use grok patterns to parse the logs. Here is the config sample I use:

<pre style="background-color:#ffffff;">
<span style="color:#323232;"># bind locally to ingest syslog messages
</span><span style="color:#323232;">[[inputs.syslog]]
</span><span style="color:#323232;">   server = "udp://<ipaddress>:6514"
</span><span style="color:#323232;">   syslog_standard = "RFC3164"
</span><span style="color:#323232;">
</span><span style="color:#323232;">[[processors.parser]]
</span><span style="color:#323232;">  parse_fields = ["message"]
</span><span style="color:#323232;">  merge = "override"
</span><span style="color:#323232;">  data_format = "grok"
</span><span style="color:#323232;">  grok_patterns = ["%{HTTPD}", "%{GEMINI}"] # this must reference the name from grok_custom_patterns
</span><span style="color:#323232;">  # format; PATTERN_NAME GROK_PATTERN…
</span><span style="color:#323232;">  grok_custom_patterns = '''
</span><span style="color:#323232;">HTTPD ^%{HOSTNAME:httphost} %{COMBINED_LOG_FORMAT} (?:%{IPORHOST:proxyip}|-) (?:%{NUMBER:proxyprot}|-)$
</span><span style="color:#323232;">GEMINI ^(?:"(?:gemini://%{HOSTNAME:gmihost}(:%{NUMBER:gmiport})?%{NOTSPACE:request}|%{DATA:raw_request})" %{NUMBER:response} %{NUMBER:bytes}|%{DATA})$
</span><span style="color:#323232;">  '''
</span><span style="color:#323232;">
</span><span style="color:#323232;"># send parsed logs to influxdb
</span><span style="color:#323232;">[[outputs.influxdb]]
</span><span style="color:#323232;">  urls = ["http://localhost:8086"]
</span><span style="color:#323232;">  database = "telegraf"
</span>

Telegraf supports logstash core patterns, as well as its own custom patterns (like %{COMBINED_LOG_FORMAT}).

You can then query your influxdb using the fields extracted from these patterns:

<pre style="background-color:#ffffff;">
<span style="color:#323232;">> USE telegraf
</span><span style="color:#323232;">> SELECT xff,httphost,request FROM syslog WHERE appname = 'httpd' AND verb = 'GET' ORDER BY time DESC
</span>

wgs, 11 months ago

I use vis to write code, ^Z and make/mk to build the project. Most of the debugging is done with valgrind and eventually gdb though my use is very limited.

When I work on manpages, I use wendy to automatically preview manpages everytime the source file changes.

wgs, 11 months ago

I use Quake 3 characters names: doom, crash, sorlag, razor, bitterman, xaero, …

wgs, 11 months ago

That’s right ! It uses BusyBox as its userland which was my main problem with it (though you can easily install GNU coreutils).

wgs, 11 months ago

I only learnt about it today, so I couldn’t check it. I have this project of building my own distro using musl and a non GNU userland, and it is a very annoying process, so I felt like I should share this one.

wgs, 11 months ago

That’s kind of the point though, as it’s now used as a base for many containers ;)

wgs, 11 months ago

It’s not about GNU being wrong or not, it’s about having the choice.

wgs, 11 months ago

I already used alpine for a few years, before containers were a thing. I heard about it exactly because it was advertised as a distro without GNU components, which was revolutionary at the time.

You sound weary with that kind of comment, I wonder what bother you so much about seeing a new distro pop up ?

wgs, 11 months ago

Arguing over licences to judge how much a piece of software is worth is sterile IMO.

If you personally cannot use software that’s not GPL’d, then it’s fine. But there’s no need to sound condescending like this, it brings absolutely nothing to the table. This could only result in a flame war (and it already is unfortunately, seeing the comments below), which is kind of sad.

So yeah, no prob mate, this is not for you, we get it. See you on the next thread 🫡

wgs, 11 months ago

Yeah it’s definitely young and not for everyone. But you gotta start somewhere ! I do agree that the “shortcomings” are not explicitly defined, but rather implied in the FAQ.

wgs, 11 months ago (edited 11 months ago)

Short answer: Don’t bother, it’s too complex to setup (unless your app is HTTP or supports the PROXY protocol). You better read your proxy logs instead.

Long answer: What you want is called “IP transparency” and require your proxy to “spoof” the IP address of the client when forwarding packets to the remote server. Some proxies do it (Nginx plus, Avi Vantage, Fortinet) but are paid services. I don’t know for free solutions as I only ever implemented it with those listed above.

This require a fairly complex setup though:

0. IP address spoofing

The proxy must rewrite all downstream request to spoof the client IP address, making it look like the traffic originates from the client at the TCP layer.

1. Backend server routing

As the packet will most likely originate from random IP on the internet, your backend server must have a way to route back the traffic to the proxy, instead of it’s default gateway. Otherwise you’d implement what is called "Direct Server Return*, which won’t work in your case (packet will be dropped by the client as originating from your backend server directly, and not from the proxy).

You have two solutions here:

• set your default gateway to the proxy over its VPN interface (don’t do that unless you truly understand all the implications of such a setup)
• use packet tagging and VRF on the backend server to route back all traffic coming from the VPN, back to the VPN interface (I’m not even sure this would work with an IPsec VPN though because of ACL…)

3. Intercept and route back return traffic

The proxy must be aware that it must intercept this traffic targeted at the destination IP of the client as part of a proxied request. This require a proxy that can bind on an IP that is not configured on the system.

So yeah, don’t do that unless you NEED to do that (trust me as I had to do it, and hated setting it up).

Edit: apparently haproxy supports this feature, which they call transparent mode

wgs, 11 months ago

Setting the default gateway to the VPN has many implications that you must take into account before doing it:

• you need to allow ALL traffic through the VPN ACL, which nullify the concept of ACL as a security measure.
• it breaks the VPN as the encapsulated packets cannot reach the other site. You need a /32 route to the other site to keep the VPN up.
• it will route ALL the internet traffic from this host through the VPN, and the internet access of the other site.
• it could break access to LAN of the server, so you might need to set your local routes manually.
• it can let your server access the LAN of the remote server, this leaking local networks.

A better option would be to use VRFs to route back traffic coming through the VPN back to it.

wgs, 11 months ago

I think you meant to reply to another comment. I never talked about setting up NAT rules, neither source, nor destination.

The proxy is responsible for responding with the correct IP address as it terminates the connection. Setting up NAT rules is not needed.

wgs, 11 months ago

This is only true if the proxy can understand the application layer of the backend (eg. HTTP). For TCP/UDP based proxy, you only get “X connected to Y” type of logs, which isn’t very useful to debug an application.

wgs, 11 months ago

Depends on the service. What application are you running on the backend server ?

wgs, 11 months ago

For TCP/UDP traffic, you’d just move the problem on another box. The application logs would report connections from 127.0.0.1 (the local proxy), and not the client IP.

wgs, 11 months ago

You should never expose a DNS server publicly

Why ?

wgs, 11 months ago

Windows does DNS cache by default, so it could be that many domains are still in your local cache. First change your DNS settings, then clear the cache with ipconfig /flushdns.

wgs, 11 months ago

tl;dr: attackers use open recursive DNS resolvers to amplify DDoS attacks.

Thanks for the link, I didn’t know about this technique. It only applies to recursive DNS though, not authoritative ones.

wgs, 11 months ago

jami.net perhaps ? I’ve just heard about it and didn’t try it myself, but it’s kind of appealing to see a new decentralized messaging app (I wish tox succeeded in this field, but hey…).

wgs, 11 months ago

It’s more about trust, than security. When you use a specific distro, you only have to trust the distro packagers. These packages are reviewed by multiple persons, tested thoroughly and (usually) built in a reproductible way. The packagers are usually different from the developers, so they can also review the code itself and eventually patch issues if needed to be in line with the distro’s ideology.

With flatpak, snap and friends, anyone is a potential packager, so for each software you gotta trust this single entity, which is usually the developer itself.

wgs, 11 months ago

Subscribe to as much shitty free services, mail lists and commercial ads as you can. If someone ever gets the domain, they’ll receive so much spam they won’t bother reading them.

Source: I bought such a domain myself, and I have no idea what the previous owner was up to… I received delivery reports from a beer company in Iceland, password reset requests from like 500 Instagram accounts, and many other emails like that.

However, the domain is totally “sane” and not blacklisted anywhere. I have no idea how that’s possible that my emails get through spam filters given it’s past ^^

wgs, 11 months ago

I use stagit. It runs whenever I push code to a repo, and then serves everything as static HTML pages.

It only provides a web interface for git repos though, and for the master branch.

wgs, 11 months ago

ELI5

So it’s saturday afternoon, a very hot one, so you ask your daddy for an ice cream (hosted service). The shop you go in is very bizarre though, as there is one vendor (TCP port) for each flavor (docker service/virtualhost). But it’s tricky because they’re all roaming in the shop, and you don’t know who’s responsible for each flavor. Your dad is also not very comfortable paying these vendors directly because they only accept cash and do not provide any receipt (self-signed certificate/no TLS).

Hopefully, there is the manager (reverseproxy) ! This girl is right where you expect her: behind the counter (port 80/443), accept credit cards and has a receipt machine (Domain name + associated certificate). She also knows everyone on her team, and who’s responsible for each flavor !

So you and your dad come to see the nice lady, ask for a strawberry + chocolate ice cream, and pay her directly. Once done, she forwards your request directly to the vendors responsible for each flavor, and give you back your ice cream + receipt. Life is good, and tasty !

wgs, 11 months ago

That one is easy ! Because in a few years (remember, you’re 5), you’ll be a scout ! And to collect a few dollars for your summer camp, you’ll sell pastries to the neighborhood. It’s easier than ever because it’s 2030, and everyone can just order the pastries on your website, and pay online. All you have to do now is hop on your bike, and deliver the pastries (network connections) to your neighbors (online servers). So you grab the first package, and read the label on it:

• Mrs. Britneak

And that’s it ! You have no idea who this person is, or where they live ! So you call out your leader (DNS server):

• Hi Mr. Leader !
• … (nobody ever get my UDP jokes)
• So I got this package to deliver to mrs. Brtineak. But I don’t know where she lives
• Oh sure, let me lookup the register (zone file). Hold on for a sec… Alright, she’s here: 62.644888, -160.194309

And then he hangs up immediately (this is UDP, remember?).

You write it down (local caching DNS server), and look it up. You’re a scout, so you’re trained to read and find GPS coordinates. You go there in a few minutes and deliver the package in time ! Mrs Britneak is happy, and you go on to the next package:

• Mr. Tomburgh

Time to call leader again !

wgs, 11 months ago

Please do ! Networking is beautiful and people need to know it !

wgs, 11 months ago

Tmux is no different from a terminal app that split the screen in terms of “multi window” functionality. However it’s not a graphical software, so you can start it remotely (eg. over ssh), and detach/reattach to it later without loosing what you where doing.

wgs, 11 months ago

That’s your use case, but you could also want to share a picture with your family, or some confidential logs with a collegue or support team. However, I wouldn’t trust any online service for this use case though. If some information is confidential, you should encrypt it yourself, share it with your peer (you could use a pastebin), then share the key over a trusted channel that’s different from this pastebin.

wgs, 11 months ago

I have a single database server because I can’t afford two servers with high storage. The servers that need access to it connect over wireguard VPN. This is slow as f**k don’t do that.

wgs, 11 months ago

Deb support will come later, but:

If the same piece of software exists in the Ubuntu repository and the snap store the new store will only make it possible to install the snap version.

So the title is on point IMO.

wgs, 11 months ago

Crux user here. I like the port tree system and simple package building recipes. It’s also a distro that kept things very simple over the years despite the rise of dbus and systems. Also the mascot.

wgs, 11 months ago

1. Definitely Ubuntu, it’s the most user friendly for people coming from other OSes
2. I’d say Ubuntu again, or maybe Debian. You built up skill and learnt the distro so you want to use something you know for work. On your personal computer, try other ones. I personally picked Arch at this time (around 2012), which helped me “understand” how the OS works, rather than simply use it. I reinstalled it quite a few times and broke the system a lot.
3. Any distro with a simple package management system. My personal choice goes to crux, but it’s very barebones. NixOS or Gentoo would be fine too The point here is to learn how to build packages by building them yourself, and I feel like the “big” packages managers (apt, yum, dnf) are too complex for that. They also decorelate runtime libraries from headers files, which is a pain to work with as a développer IMO.

But it’s just my personal experience, many new distro popped up since then. Also for reference, I’ve been using Linux for 12 years now, and I run Crux on my desktop, Ubuntu at work, and OpenBSD on my servers.

wgs, 11 months ago

Thanks for the feedback. I didn’t try it because I didn’t want to buy Sailfish OS (again…) just to end up with a broken phone, and rollback to android, especially as it breaks the warranty. I figured I could just wait for the next update for these issues to be fixed, but they never came and I simply did not bother getting the test image at this point.

Would you have a link to the patch for the camera ?

wgs, 11 months ago

This matrix is for features supported via ANSI escape codes.