/make/ - Creative

World made of things

Want your event posted here? Requests accepted in this /meta/ thread.

Max message length: 5120

Drag files to upload or
click here to select them

Maximum 5 files / Maximum size: 20.00 MB

More

(used to delete files and postings)


Open file (86.48 KB 1438x846 nibs.png)
NIBS imageboard client/server Anonymous 04/28/2020 (Tue) 19:30:44 No.63
There is a concept that's been hovering around for a while, but seemingly nobody has made a serious attempt at it. That is, a desktop client that can read imageboards. There's a lot of advantages to this, but you can take it a step further and reach a whole another level by also making a server that's designed to co-operate with the client. This thread is for me ideaguying and probably posting actual progress on such a system (from here out named NIBS for "N-ImageBoard System"). NIBS is 2 different things, a desktop client for browsing imageboards, and an imageboard server that's designed for the client. The basic idea behind it's operation is that whenever you get information from the server, you cache it locally on your computer, and when you view a page it's loaded from the cache, and you only ask the server if anything has happened after that. This results in colossal speed gains for the user and bandwidth savings for the server, the experience is better for the user and the server costs are much lower for the owner. The auto-update system will also be able to tell you everything about every thread/board you have open/watched across the entire webring, instantly as soon as something happens anywhere. By default, NIBS imageboards will be impossible to access directly from a web browser, and the NIBS client can't directly access traditional website imageboards. Instead both the NIBS client and server will be split into 2 parts; one for the NIBS core system, and another optional "plugin" that can translate to/from web protocols. The plugins will allow the NIBS client to read traditional imageboards, and the NIBS server to be accessible from a web browser like any other imageboard (some features will be disabled however because web browsers can't do certain things that NIBS client can, and traditional servers don't support certain NIBS server features). On the server side, this also allows you to detach the imageboard from the web by removing the plugin, thus only people with the NIBS client can access it. NIBS will have a "webring" by design, in fact the whole system revolves around a webring-like operation. It works similarly to the current webring except servers only store links to other imageboards, they don't share board lists. Instead the user's client will retrieve all the board data from all the different servers, and stores them locally. It will also be able to include the current webring boards. Pic related is a shitty graph that attempts to roughly describe how NIBS server and client communicate and how the "web plugins" play in. (just replace "NC" with "NIBS") Nchan ( >>4 ) will likely be "cancelled" and replaced with NIBS (that's where the N comes from). The web translator plugin should eventually be able to provide a website that basically looks like Nchan anyway though. However finishing NIBS will take a much longer time than it would to finish Nchan. NIBS is the true true thing I've really want to make, more so than a normal imageboard, and more so than just a handmade server. I just didn't have enough understanding of networking to think about this seriously. TL;DR A desktop program for browsing imageboards. A new imageboard engine. A completely new model of operating which has huge advantages.
Why? There's massive advantages to NIBS compared to a traditional imageboard. Here's some of them: - after the first time viewing a page, loading the page later will be instantaneous, and only requires a small update to check if anything has changed. - you can view pages if the server is down, and even if you're not connected to internet. You can also opt to keep deleted content. - both the server and the client will be a LOT lighter, will use a very small amount of bandwidth compared to normal imageboards, and will be much more CPU/RAM efficient than any normal imageboard can hope to be. - the webring is on your side, so even if a website goes down, you'll still see all the websites and board lists just the same (currently you'll have to go to another website to see the webring board list, and you'll have to remember what the URL for the another one is). - you'll be able to add webring sites yourself, and favorite/remove boards as you please. - you'll be instantly notified when a watched thread is updated anywhere in the webring, and you can switch to it instantly without loading anything. - all the websites will work in a consistent way. - you can have all the features of javascript and much more, yet there's no risk of malicious scripts or datamining because all the code is on your side and the only data that's sent is minimal requests for posts and such. You can also customize the scripts to your preference. - the frontend will be incredibly customizable. - a NIBS imageboard can become hidden from most people simply by not using the web plugin, since people will not be able to view it through a web browser. - the client will store server IPs as it finds them, thus the users are less affected by kikery on the DNS level. DNS is not even necessary since the imageboard can let itself be found through the webring. - all your settings and themes and customizations will be with you. They won't be reset just because you move to another server, and you can even easily port all your settings between computers. - you can make your imageboard hidden from the average normalnigger simply by not using the web plugin. - because it's a desktop program, it will be possible to eventually add support for IPFS or other distributed functionality that does not rely on central servers, along with OpenNIC and whatever interesting projects. Additionally, the server will be absolutely foolproof to set up. This is more about my personality as a developer than about the design of NIBS, but I want it to be possible to start the server just by downloading it and then running the program. I HATE, I FUCKING HATE configuring shit, that's just bad software design as far as I'm concerned, so my goal is for both the server and the client to be compiled just by calling the compiler (or a .sh/.bat file which is provided with the code). No make systems or python interpreters or configurations or any other cancer required, just a C compiler. I also intend to make the source code as easy to understand and well commented as possible, so that people can explore the source and understand how everything works without having to worry if there's some botnet inside.
The obstacles It's not all flowers and sunshine though. From what I can think of, there's 4 primary technical complications to making this. First off it's going to be hard to embed files. It's very difficult to add comprehensive support for image formats, formats like PNG or god forbid videos are so complex that there's no way I can do it myself, and libraries like STB don't support the full image format. It'll basically all depend on my ability to find libraries that can decode those formats and give me the means to draw it on the screen (or play audio). I have also never rendered a video onto the screen, so I'm very uncertain about anything related to it (e.g. whether it requires hardware acceleration). To support everything that browsers do, you need a lot of libraries and a lot of spaghetti to put it all together, and I haven't had good experiences trying to use other people's libraries. Worst case scenario: images will be supported to the extent that stb_image can handle (I've used it in the past, should mostly support png/jpg images), and you'll have to use the image viewer/video player on your computer to open anything else. Second is text rendering. It's basically impossible to read font files yourself so that requires a library. As for the rendering itself, it's easy to draw text by using bitmap character sprites, but the problem with that method is you can't do fine position adjustments with bitmaps, which starts to look janky when the text is small. STB has a font library which should be able to load font files and make bitmap sprites from the characters. Worst case scenario: I can just use custom bitmap fonts which I've done before, though it's harder to use different font sizes so they may look too big/small. I'm almost 100% sure it's be possible to use STB to make bitmap character sheets out of regular fonts, it just might not look clean. Third is text input. I don't have any idea how you're supposed to make a text input box that supports esoteric keyboard mechanics like typing Japanese characters. It's also hard to support different keyboard languages in general. Many games support typing unusual languages so there must be some well known way to do it. There might be something in the operating system, but it might end up being hacky and weird and have bad cross-platform consistency. I haven't researched this topic much. Worst case scenario: you can only type english letters and common symbols like punctuation. Fourth is DDOS protection. Since it's not a web program, I don't know if it's possible to use anti-DDOS services like cloudflare. I have basically 0 experience on this topic or how it works so it's hard to say whether it will be a problem or not. Worst case scenario: you have to hope nobody aims their autistic rage and money at you, or just wait it out if it happens (nobody has the money to DDOS forever).
=The plan I don't necessarily want to talk about plans or ETAs or anything like that, but this thread probably makes NIBS look like some super huge and serious project (it's not, it's just one of many things I'm working on for fun), and because of what's happening around imageboards recently I feel like someone might come here and start anticipating this project's release. If someone wants an imageboard then they shouldn't wait for this. It might take a long time to make this, it might even take years, but it might also take a month, it might be that I get killed by corona because retards don't understand what social distancing is, I just can't predict. I could release Nchan much sooner than NIBS, but the current state of imageboards makes me feel that something like NIBS is much more important, plus this is what I eventually want anyway. Currently NIBS is just a bunch of design ideas and some preliminary C code. I started making an imageboard engine in node.js ( Nchan >>4 ) so I technically have a lot of the server code figured out, it just needs to be ported to C. And as that implies, NIBS will be programmed in C because it's my preferred language. The whole project in it's entirety will of course eventually be FOSS (probably AGPL) when it's ready to be used, but I generally won't allow code contributions outside small or specific cases. I want to make sure the codebase is clean and works the way I want, otherwise it'll become harder and less fun to maintain. When the source is ready to be released I'll definitely appreciate help with getting image/video format support for example, or bug fixes.
>>66 You can take a look at BUMP for ideas about how I addressed all the differing IB JSON formats Anon. Might give you an idea or two.
Open file (132.74 KB 330x361 1585677404668.png)
>fatchan is out I think at this point I've seen people forced to migrate enough times that something has to be done about that. This is something I've thought about before a little bit, but not very seriously because I haven't seen good examples of distributed systems, particularly the ability to moderate them. I want to integrate some kind of backup mechanics and fallbacks to NIBS to at least alleviate this problem, and the solution I'm currently thinking of is a type of federated server network. I suck at terminology so I'm just going to invent a new term by calling it KSM (Key Server Mirroring) for now. The idea is that you have one single "key" server, and it can share all of it's content with "mirror" servers in real time. The software on both is identical, the only difference is how they handle input/output based on what mode they're on. Anyone can set up a mirror server and mirror any key server that they want, and the imageboard will be accessible through any mirror server as well as the key server. Mirror servers cannot "commit" new posts though, only pass it on to the key server for confirmation. A key server can mark mirror servers as it's "successors". In the event that the key server goes down for ~24-48 hours (debatable), the highest priority successor mirror server will switch itself to key server mode, and all users and mirror servers will automatically switch to that as the new key server. Since it's mirroring all the content, all the posts and files and such will already be there, and the migration process is completely seamless and automatic for everyone (ignoring possible complications with moderator accounts). The key server can also voluntarily release itself from duty and automatically assign a given mirror server as the successor. Any mirror server will be able to turn into a key server at any point in time, but unless it's the highest priority successor of a key server that's been down for a while, it will create a new separate forked imageboard and nobody will switch to it automatically. While mirror servers cannot host their own boards, I'm thinking about treating /servermeta/ or something as a special board that will never be mirrored, instead it's something that will always be tied to the server and thus mirror servers can have their own as well if they want. I think in some sense it's similar to Robi's Final Solution, except only one server is in control at one time, there's no different types of views into the same board. In terms of implementation, there shouldn't be huge differences to how I planned to do things anyway, mirroring is just a slight expansion of the auto-update and "tell me what happened after X" mechanics, and keeping track of servers and switching between them is trivial. ALSO it might be possible to turn this system into a distributed system with small tweaks (basically any mirror server will be able to "commit" content, and they'll also accept content from any other mirror server), though I'm not sure how moderation would work with that, and you'd have to change the way post numbers work. Speaking of moderation, I wonder if you could use some kind of irreversible cryptography to create moderator account passwords that work across servers without anyone being able to know the original password (except the server you send the password to).
>>67 I've heard of that once upon a time but it's not a name that makes it easy to search for. I could look into it if you gave a link. Reading traditional imageboards is pretty low on the priority list so I won't need it for a while though.
I looked into input handling, and it turns out there's a special window event (WM_CHAR) that fires when a character is input, rather than when a keyboard key is pressed. That even handles dead keys on my keyboard correctly, I think it might also output the correct character codes for things like Russian keyboards, I want to get one just to try it. But apparently if you want to input Japanese or similarly complex language that there's no direct keyboard for, you'll need a special mechanism to handle it. Windows has something called IME which does it, but it looks pretty annoying to use. I tried the stb font and image libraries and they both seem to work well. I didn't finish making a text drawing system though so I don't know how bad the letter spacing inaccuracy problem will be. I'm the kind of person who doesn't give a shit how hard something is. If I think something needs to be done, then I'll just seriously approach doing it instead of going "it's too hard" and giving up. And one of the things I think needs to be replaced the most is the entire fucking internet. There's many kinds of websites I'd like to make, and instead of making them as normal websites, I begun wondering if it would be possible to make something similar to the web except using some of the basic ideas behind NIBS. It's all completely hypothetical, but currently what I'm thinking of is that instead of loading a bunch of HTML files all the time, it operates by loading a bunch of "templates" and "content", and both are cached by the client so they won't need to be loaded a second time. The client then asks for updates for the templates and new content that might have happened whenever appropriate, just like NIBS would. If it seems like it'll work out, I might do that and then make the imageboard part of NIBS on top of it instead of making NIBS it's own thing. I wanted to take screenshots of different websites and think about what kind of templates and mechanics would be needed to make them, but I don't have an image editor that I can use to easily make that kind of prototypes, so now my focus is shifting to making an image editor. There's seriously so many things that the world needs that it's impossible for me to make them all, it's not even easy to list all the things. There's effectively nothing that I use on the computer or the internet that I wouldn't prefer a better alternative to, including of course the OS and internet themselves. Making an OS is something like a dream project among dream projects for me, but I have no idea about anything related to the internals of operating systems so I can't even hypothesize about the mechanics or how to build anything, I doubt I'll ever actually make one.
>>75 might i suggest you use std::cin instead of tying yourself (and your users) down to a POS like Windows? could be a good idea.
>>77 Being bound to a certain language is not something I plan on doing, even if it would make things easier. I will support Linux though, I just don't want to add unnecessary walls in front of me right now.
>>79 Suit yourself ofc, but a) the standard console in is a tried-and-true part of a tried-and-true language. Use scanf() if you'd prefer, same story. Tying your system to Windows is a far more risky decision in so many ways.
>>86 I don't really understand why you think the program is going to be tied to Windows somehow. There's a couple ways to design cross-platform code, and the one which I use is to have an abstraction layer where I define my own versions of all the OS-specific functionalities. I just need to make a Linux version of that file which is less than 3000 lines of code at this point including networking, and I'll never need to care about the OS anywhere else in the codebase again.
>>95 >I don't really understand why you think the program is going to be tied to Windows somehow. oh, just b/c raisins somehow. https://docs.microsoft.com/en-us/windows/win32/inputdev/wm-char
Some links that may or may not be of interest: https://github.com/DogMan8/CatChan https://github.com/catamphetamine/captchan https://github.com/siavash119/qtchan If I think of any others, I will post them.
Well anon, the whole project looks pretty fucking dope. Especially the "required client" part. I'd be so happy if I could purge the browser from my computer. As cool as it sounds, I have a few comments though: >C Unless you wrote an entire OS in it and the thing is stable (you didn't), you shouldn't use it for networking. You'll end up with so many accidental leaks that you'll want kys. You should definitely use something else or at least get a library for all the string operations. >will use a very small amount of bandwidth compared to normal imageboards Wdym? You want to compress all the traffic? >Interface I suggest going with SDL, since imageboards usually contain a variety of media (formatted text, audio, video) and I think it would be easier to make an IB client with a shim that allows you to draw everything instead of fighting with predefined GUI systems. Text user interfaces are out of the question when it comes to imageboards. I personally tried that once (https://gitgud.io/fixerb/tibb) and after a few months of terminal quirks and glitches I realized that imageboards just aren't meant for the terminal. >DDOS protection Eh, you can always do rate limiting or enable captcha when the flood gets too big.
Open file (1.85 MB 1020x638 fnw.webm)
>no posts in a week >log in to make sure the board isn't in claims >someone posts Does logging in promote the board somehow? I feel like this has happened more than once. >>125 >I'd be so happy if I could purge the browser from my computer It won't be able to connect to normal websites though, that's impossible without some kind of browser context running in the background at which point you're just making a regular web browser. It's possible for some imageboards because of the special module whose sole purpose is to translate the imageboard pages/forms. >you shouldn't use C I don't see why not, there's only minor inconveniences with C that I care about. I've mostly already made my own string libraries and they're fine. >"use a very small amount of bandwidth" >Wdym? You want to compress all the traffic? I think I already explained it but the whole idea with NIBS is to completely change how pages and resources work. With normal websites you have to load HTML pages and resources over and over every time you view them, there's kind of caching but it's very primitive and unreliable. With NIBS pages and images will be cached in the client and never need to be reloaded from the server again (unless you clear the cache or the relevant part of it expires). If you view the same page again, it doesn't necessarily load anything at all from the server, all it does is ask whether something has changed, and it either receives a "no" reply, or the new contents and only the new contents (e.g. a new post in a thread). The server can also send data in a binary format instead of parsing it from/to some kind of HTTP+HTML text document, you may think it doesn't matter because "it's just text", but as soon as you load a thread with thousands of posts, the HTML alone starts to add up to megabytes of data that doesn't really need to be there. For example if I remove the html from your post and leave only the actual information it contains, the size is literally cut in half and a lot of your post is text. A post like >>100 on the other hand becomes about 8 times smaller. >SDL >predefined GUI systems My plan was to not use anything, I don't want to make a program by gluing together huge bloated systems that do who knows what behind my back and may have issues that I can't fix. I'm already able to draw basic UIs and pages and text with my own stuff, webm related is another program I made for myself with 0 libraries. The only problem is decoding video/image formats since there's way too many and they're way too complicated for a single person to understand let alone make decoders for. If SDL has some separate components for handling videos/images then I'll consider using those, I just don't want my entire program to be a slave to it. I may also look into what MPCHC and irfanview and similar are doing when I get there since those programs support a lot of media file formats. >you can always do rate limiting or enable captcha when the flood gets too big If it was that easy cloudflare wouldn't exist. You can still overwhelm the server by sending millions of requests per second from a huge botnet that you paid the Chinese $20 for.
>>126 >and they're fine. <le famous last words maymay no, they're probably not. proper array management directly using pointers is notoriously difficult to get right, especially in the presence of exceptional conditions. given that hand-spun strings are probably the number one online security hole in existence, i'd recommend you stick with a library for strings. or better still, just use C++ strings and the RAII idiom.
>>127 It's literally just about moving chunks of memory from A to B, index*itemSize, is this considered to be some rocket science to modern programmers? I don't understand why there's supposed to be a problem, if that's too complicated then you won't be able to do anything because that's about as basic as it gets. I'll hear you out though if you can point out examples of where a problem occurs. >use C++ RAII 1. that's not even related to your objection 2. kys
>>128 >1. that's not even related to your objection Actually, it exactly is. Perfect resource management in the presence of exceptions is trivial using RAII. Just don't leak in the first place. I doubt you can say the same of your string 'library'. >kys lolnou.
>>129 Well you use whatever the C++ people tell you to then, I'll do what works for me. I get the feeling you won't accept anything short of "wow I'm going to use [C++ feature] now".
>>130 I try to use what I've proved both works, is as reasonably secure as understood atm, is stable, and highly performant. Hand-rolled C solutions rarely exhibit any of these qualities, and certainly not anywhere near the period of rollout. Using the C standard library instead would go a long way to helping you out here. OTOH, the C++ standard library typically exhibits all these qualities, and basically right out of the gate. It's not at all difficult to get things right with basic C++ skills. And it often translates into industrial-scale solutions with little to no modifications to a simplistic approach. Having the attitude that you understand things much better than seven men who can answer wisely is surely the sign of a fool.
>>131 >C++ code is automatically super industrial grade this and that You sound so pretentious, using a lot of smart sounding marketing words doesn't make you "seven wise men". I've heard several people guaranteed much more experienced than you (Mike Acton, Sean Barrett, Jonathan Blow, Casey Muratori, along with other developers in the handmade community) say that C is fine and that most of C++'s standard library is complete shit and claim that many other people have similar opinions. I don't necessarily subscribe to that as line of thought as much as they do nor "avoid C++ because it's bad" because I don't have much experience with it, I just don't use it because it doesn't really give solutions to the problems I've had with C. You haven't seen any of my code nor what I'm doing but you've already decided that it's not good enough because I didn't use C++ features to make it. You invented a problem (strings are hard) and berate me for not using X to solve it. There's this certain class of people who are REALLY big fans of C++ and take issue with anyone who doesn't use or like it, it's very tiresome. I mean you can like and use whatever you want and suggest other people do but this whole argument is basically ideological and a complete waste of time, it's better suited to /tech/ and /g/ style boards where people argue about tools and methods more than they actually make anything with them.
>>127 >proper array management directly using pointers is notoriously difficult to get right Array management and pointers are Programming 101, and any language that holds your hands nevertheless use it in their implementation. I recommend you jay off C++ for a while and and learn how the black box actually works. You'll become a much more proficent programmer for it.
I started prototyping page templates for the web alternative mentioned >>75 and so far it's not looking too good. TL;DR of this entire post: privacy complications, in order to solve them you need to make the system much less cool. It's also not as easy to "just disable it" as it is to just disable javascript. Problem is not about implementation, it's that I underestimated how easy it is to abuse the templating system's basic functions to compromise user's privacy. It's hard to do even a basic imageboard without if/else conditions and such, for example showing a lock icon or disabling the "reply" button when a thread is locked. If conditions alone may open up a bag of privacy holes. I'm also not sure how you'd do something similar to (You)s, especially without allowing the template to use the (You) information elsewhere to do excessive things. How you'd do it is by storing special local variables based on what the server tells you. For example you make a post, the server sends you "your post was successful, posted as {123}", then there's a local "myPosts" array which receives "123", and the template checks myPosts when it parses the post reply links and adds a (You) if there's a match. But that kind of mechanic gives the server too much power, one of the main advantages of this was supposed to be that it's more private and secure since the server can only throw generic templates and contents to you and not really do anything else. Similarly how would you favorite and bookmark things, like images in a gallery or boards in 8chan? This is maybe easier to fix because the server doesn't and shouldn't know what you're favoriting at any point. But how do you stop the template from sending those values into the server, for example by pasting onto the page a 1x1 image file whose URL is compiled from all your favorite pokemon smut image IDs, and then the client requests that resource? I suppose in that particular case the favorites gallery template could just load special tracking thumbnails when you view your favorites. You can't really favorite things or show (You)s on the browser if you disable javascript and cookies either, so maybe it's just inevitable that functionality opens up more ways of tracking people. One idea could be that if a local variable is used in any way, you can only load local resources within the block that queried it. This way you couldn't link a special tracking thumbnail from your 'saved images' gallery or anything like that (though you could modify the links to lead to unique tracking pages). This would also safely allow (You)s since it doesn't need to load anything, only print plain text. Other option is to simplify the system by not allowing any kind of if/else conditions or saved variables. You could still make an imageboard, it just would lack certain details like (You)s and you'd have to use tricks to get things like thread locked icons (for example by saving the appropriate status icons as HTML in the database and just pasting them onto the template). But now the problem is that my interest in developing this whole idea goes down a lot if it's going to be that much more restricted than the web. It's getting more and more questionable about being capable of being what I wanted NIBS to be. I had already given up certain ideas I had for it because they're impossible without a Turing complete programming language with way too much power. One of those ideas was to integrate oekaki-related features more than imageboards typically do. Since NIBS will be a dedicated program where none of the functionality is loaded from a server, it can basically do anything without restriction. The last resort is to implement an actual scripting language that's applied on top of the page separately, similar to javascript. That would make it simple to disable, but it's basically the "I give up" option, I don't want this system to be something where you need to disable things to feel safe. I did consider plugins too, like you could add oekaki and other features into your imageboard by telling the users to install a plugin that does it. But that's pretty extreme and dangerous in a different way, it's possible for such a plugin to accidentally have a more serious security hole that compromises your entire computer. And all this is just what I discovered from prototyping imageboard templates, who knows what other problems I haven't realized yet. I'll most likely just give up on the idea and make NIBS as originally intended; as it's own thing. Not having to work around tweaky limitations and give up on ideas because of it sounds much more exciting. The biggest downside of NIBS is that you can't expand the network with other interesting pages, it would be really cool if the client was more than just a special imageboard.

Report/Delete/Moderation Forms
Delete
Report

Captcha (required for reports and bans by board staff)

no cookies?